Non-invasive quantitative imaging biomarkers of atherosclerotic plaque biology

ABSTRACT

Systems and methods for analyzing pathologies utilizing quantitative imaging are presented herein. Advantageously, the systems and methods of the present disclosure utilize a hierarchical analytics framework that identifies and quantify biological properties/analytes from imaging data and then identifies and characterizes one or more pathologies based on the quantified biological properties/analytes. This hierarchical approach of using imaging to examine underlying biology as an intermediary to assessing pathology provides many analytic and processing advantages over systems and methods that are configured to directly determine and characterize pathology from underlying imaging data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of the co-pending U.S.application Ser. No. 16/203,418 filed Nov. 28, 2018, which is acontinuation-in-part of the co-pending U.S. application Ser. No.14/959,732, filed on Dec. 4, 2015, which claims priority to U.S.Provisional Application Ser. No. 62/205,295, filed on Aug. 14, 2015;U.S. Provisional Application Ser. No. 62/205,305, filed on Aug. 14,2015; U.S. Provisional Application Ser. No. 62/205,313, filed on Aug.14, 2015; U.S. Provisional Application Ser. No. 62/205,322, filed onAug. 14, 2015; and U.S. Provisional Application Ser. No. 62/219,860,filed on Sep. 17, 2015.; U.S. application Ser. No. 16/203,418 alsoclaims priority to U.S. Provisional Application Ser. No. 62/676,975,filed on May 27, 2015; and U.S. Provisional Application Ser. No.62/771,448, filed on Nov. 26, 2018.

GOVERNMENT RIGHTS IN THE INVENTION

This invention was in part made with government support under NationalScience Foundation SBIR Award 1248316 and National Institute of HealthSBIR Award R44 HL 126224-01A1. The government may have certain rights inthe invention.

BACKGROUND OF THE INVENTION

The present disclosure relates to quantitative imaging and analytics.More specifically, the present disclosure relates to systems and methodsfor analyzing pathologies utilizing quantitative imaging.

Imaging, particularly with safe and non-invasive methods, represents themost powerful methods for locating the disease origin, capturing itsdetailed pathology, directing therapy, and monitoring progression tohealth. Imaging is also an extremely valuable and low-cost method tomitigate these human and financial costs by allowing for appropriateearly interventions that are both less expensive and disruptive.

Enhanced imaging techniques have made medical imaging an essentialcomponent of patient care. Imaging is especially valuable because itprovides spatially- and temporally-localized anatomic and functionalinformation, using non- or minimally invasive methods. However,techniques to effectively utilize increasing spatial and temporalresolution are needed, both to exploit patterns or signatures in thedata not readily assessed with the human eye as well as to manage thelarge magnitude of data in such a way as to efficiently integrate itinto the clinical workflow. Without aid, the clinician has neither thetime nor often the ability to effectively extract the informationcontent which is available, and in any case generally interprets theinformation subjectively and qualitatively. Integrating quantitativeimaging for individual patient management as well as clinical trials fortherapy development requires a new class of decision support informaticstools to enable the medical community to fully exploit the capabilitiespossible with evolving and growing imaging modalities within therealities of existing work flows and reimbursement constraints.

Quantitative results from imaging methods have the potential to be usedas biomarkers in both routine clinical care and in clinical trials, forexample, in accordance with the widely accepted NIH Consensus Conferencedefinition of a biomarker. In clinical practice, quantitative imaging isintended to (a) detect and characterize disease, before, during or aftera course of therapy, and (b) predict the course of disease, with orwithout therapy. In clinical research, imaging biomarkers may be used indefining endpoints of clinical trials.

Quantification builds on imaging physics developments which haveresulted in improvements of spatial, temporal, and contrast resolutionas well as the ability to excite tissues with multipleenergies/sequences, yielding diverse tissue-specific responses. Theseimprovements thereby allow tissue discrimination and functionalassessment, and are notably seen, for example, in computed tomography(CT), dual energy computed tomography (DECT), spectral computedtomography (spectral CT), computed tomography angiography (CTA), cardiaccomputed tomography angiography (CCTA), magnetic resonance imaging(MRI), multi-contrast magnetic resonance imaging (multi-contrast MRI),ultrasound (US), and targeted or general contrast agent approaches withvarious imaging modalities. Quantitative imaging measures specificbiological characteristics that indicate the effectiveness of onetreatment over another, how effective a current treatment is, or whatrisk a patient is at should they remain untreated. Viewed as ameasurement device, a scanner combined with image processing of theformed images has the ability to measure characteristics of tissue basedon the physical principles relevant to a given imaging approach and howdiffering tissues respond to them. Though the image formation processdiffers widely across modalities, some generalizations help frame theoverall assessment, though exceptions, nuances, and subtleties drive thereal conclusions and until and unless they are considered some of thegreatest opportunities are missed.

Imaging in the early phases of clinical testing of novel therapeuticscontributes to the understanding of underlying biological pathways andpharmacological effects. It may also reduce the cost and time needed todevelop novel pharmaceuticals and therapeutics. In later phases ofdevelopment, imaging biomarkers may serve as important endpoints forclinical benefit and/or as companion diagnostics to help prescribeand/or follow specific patient conditions for personalized therapy. Inall phases, imaging biomarkers may be used to select or stratifypatients based on disease status, in order to better demonstratetherapeutic effect.

a. Quantitative Medical Imaging:

Enhanced imaging techniques have made medical imaging an essentialcomponent of patient care. Imaging is especially valuable because itprovides spatially- and temporally-localized anatomic and functionalinformation, using non- or minimally invasive methods. However,techniques to deal with increasing resolution are needed, both toexploit patterns or signatures in the data not readily assessed with thehuman eye as well as to manage the large magnitude of data in such a wayas to efficiently integrate it into the clinical workflow. With newerhigh-resolution imaging techniques, unaided, the radiologist would“drown” in data. Integrating quantitative imaging for individual patientmanagement will require a new class of decision support informaticstools to enable the community to fully exploit the capabilities of thesenew tools within the realities of existing work flows and reimbursementconstraints.

Additionally, quantitative imaging methods are increasingly important to(i) preclinical studies, (ii) clinical research, (iii) clinical trials,and (iv) clinical practice. Imaging in the early phases of clinicaltesting of novel therapeutics contributes to the understanding ofunderlying biological pathways and pharmacological effects. It may alsoreduce the cost and time needed to develop novel pharmaceuticals andtherapeutics. In later phases of development, imaging biomarkers mayserve as important endpoints for clinical benefit. In all phases,imaging biomarkers may be used to select or stratify patients based ondisease status, in order to better demonstrate therapeutic effect.

Improved patient selection through use of quantitative imaging couldreduce the required sample size for a given trial (by increasing thefraction of evaluable patients and/or decreasing the impact of nuisancevariables) and help to identify the sub-population that could benefitmost from the proposed treatment. This should reduce development timeand cost for new drugs, but might also result in reducing the size ofthe ‘target’ population accordingly.

Disease isn't simple, and whereas it often manifests itself focally, yetit is often systemic. Multifactorial assessment of objectively relevanttissue characteristics represented as a panel or “profile” of continuousindicators, sometimes ideally proven as a “surrogate” for a futureand/or hard to measure but accepted endpoint, has proven to be aneffective method across medicine and will do so here. Computer-aidedmeasurement of lesion and/or organ biology and quantification of tissuecomposition in first- or second-reader paradigms made possible by aninterdisciplinary convergence between next generation computationmethods for personalized diagnostics based on quantitative imagingassessment of phenotype implemented in an architecture which proactivelyoptimizes interoperability with modern clinical IT systems providespower to the clinician as they manage their patients across thecontinuum of disease severity for improved patient classification acrosssurgical, medical, and surveillance pathways. More timely and accurateassessments yield improved outcomes and more efficient use of healthcare resources, benefits that far outweigh the cost of the tool—at alevel of granularity and sophistication closer to the complexity of thedisease itself rather than holding the assumption that it can besimplified to a level which belies the underlying biology.

b. Phenotyping:

Radiological imaging is generally interpreted subjectively andqualitatively for medical conditions. The medical literature uses theterm phenotype as the set of observable characteristics of an individualresulting from the interaction of its genotype with the environment.Phenotype generally implies objectivity, namely that the phenotype maybe said as being true rather than subjective. Radiology is well knownfor its ability to visualize characteristics, and increasingly it may bevalidated to objective truth standards (U.S. application Ser. No.14/959,732; U.S. application Ser. No. 15/237,249; U.S. application Ser.No. 15/874,474; and U.S. application Ser. No. 16/102,042). As a result,radiological images may be used to determine phenotype, but adequatemeans to do so are often lacking.

Advantageously, phenotyping has a truth basis and therefore can beindependently and objectively evaluated. Furthermore, phenotyping isalready an accepted formalism in the healthcare industry for managingpatient therapeutic decisions. Thus, it has a high degree of clinicalrelevance. Finally, phenotyping is consumer relatable. This allows forboth self-advocacy as well as serving as motivator for lifestylechanges.

Early identification of phenotype based on a comprehensive panel ofcontinuous indicators rather than merely as the detection of a singlefeature would enable prompt intervention to prevent irreversible damageand death. Solutions are critical to preempt events outright or at leastto improve the diagnostic accuracy on experiencing signs and/orsymptoms. Efficient workflow solutions with automated measurement ofstructure and quantification of tissue composition and/or hemodynamicsmay be used to characterize patients at higher risk, who would betreated differently from those who are not. If we tie characteristics ofplaque morphology to embolic potential there will be huge clinicalimplications.

Imaging phenotypes may be correlated gene-expression patterns inassociation studies. This may have a clinical impact as imaging isroutinely used in clinical practice, providing an unprecedentedopportunity to improve decision-support in personalized treatments atlow cost. Correlating specific imaging phenotypes with large-scalegenomic, proteomic, transcriptomic, and/or metabolomic analyses haspotential to impact therapy strategies by creating more deterministicand patient-specific prognostics as well as measurements of response todrug or other therapy. Methods for extracting imaging phenotypes todate, however, are mostly empirical in nature, and primarily based onhuman, albeit expert, observations. These methods have embedded humanvariability, and are clearly not scalable to underpin high-throughputanalysis.

At the same time the convergence of unmet needs to achieve morepersonalized medicine while not adding cost—indeed, to better controlcost through initiatives in preventative medicine, comparativeeffectiveness, reimbursement approach, and/or avoiding rather thanreacting to untoward events present unprecedented pressures ontechnological advances that not only provide capability but to deliverit in ways that simultaneously reduce cost.

In addition to the problem of phenotype classification (classifyingunordered categories) is the problem of outcome prediction/riskstratification (predicting ordinal levels of risk). Both have clinicalutility but being the result of different technical devicecharacteristics. Specifically, one does not strictly depend on theother.

Without limiting generality, an example of phenotype classificationincluded clinical relevance is provided below:

Examples manifestations of “Stable Plaque” phenotype of atherosclerosismay be described as follows:

-   -   “Healed” disease, low response to intensified anti-statin regime    -   Fewer balloon/stent complications    -   Sometimes higher stenosis>50%    -   Sometimes higher, Deeper Ca    -   Minimal or no lipid, hemorrhage, and/or ulceration    -   Smooth appearance

Such plaques generally have a lower adverse event rate than an “UnstablePlaque” phenotype.

-   -   Active disease, may have high response to intensified lipid        lowering and/or anti-inflammatory regime    -   More balloon/stent complications    -   Sometimes lower Stenosis<50%    -   Low or diffuse Ca, Ca close to lumen, napkin ring sign, and/or        microcalcification    -   Sometimes evidence of more lipid content, thin cap    -   Sometimes evidence of hemorrhage or intra-plaque hemorrhage        (IPH), and/or ulceration

Such plaques have been reported to have 3×-4× the adverse event ratecompared to Stable phenotypes. These two examples may be assessed at asingle patient encounter, but other phenotypes such as “RapidProgressors” may be determined by comparing the rate of change incharacteristics over time, i.e., phenotypes informed by not only what isstatically present at one point in time but rather which are determinedbased on kinetics and/or how things are changing n time.

c. Machine and Deep Learning Techniques:

Deep Learning (DL) methods have been applied, with large success, to anumber of difficult Machine Learning (ML) and classification tasksstemming from complex real-life problems. Notable recent applicationsinclude computer vision (e.g., optical character recognition, facialrecognition, interpretation of satellite imagery, etc.), speechrecognition, natural language processing, medical images analysis (imagesegmentation, feature extraction, and classification), clinical andmolecular data biomarkers discovery and verification. An attractivefeature of the approach is its ability to be applied to bothunsupervised and supervised learning tasks.

Neural Networks (NN), and the Deep NN approach, broadly includingconvolutional neural networks (CNNs), recurrent convolutional neuralnetworks (RCNNs), etc., have been shown to be based on a soundtheoretical foundation, and are broadly modelled after principlesbelieved to represent the high-level cognition functions of the humanbrain. For example, in the neocortex, a brain region associated withmany of cognitive abilities, the sensory signals propagate thru acomplex local, modular hierarchy which learns to represent observationsover time—a design principle that has led to the general definition andconstruction of CNNs used for i.e. image classification and featureextraction. However, it has been somewhat of an undecided question whatare the more fundamental reasons for the superior performance of the DLnetworks and approach compared to frameworks with the same number offitting parameters but without the deep layered architecture.

Conventional ML approaches to image feature extraction and image-basedlearning from raw data have a multitude of limitations. Notably, spatialcontext is often difficult to obtain in ML approaches using featureextraction when features are at a summary level rather than at the 3Dlevel of the original image, or when they are at the 3D level, to theextent that they are not tied to a biological truth capable of beingobjectively validated but just being mathematical operators. Use of rawdata sets that do contain spatial context often lack objective groundtruth labels for the extracted features, but use of raw image dataitself include much variation that is “noise” with respect to theclassification problem at hand. In applications outside of imaging, thisis usually mitigated by very large training sets, such that the networktraining “learns” to ignore this variation and only focus on thesignificant information, but large training sets of this scale are notgenerally available in medical applications, especially data setsannotated with ground truth, due to cost, logistics, and privacy issues.The systems and methods of the present disclosure help overcomes theselimitations.

d. Example Application: Cardiovascular Measures Such as Fractional FlowReserve (FFR) and Plaque Stability, High-Risk Plaque (HRP) or Erosion:

New treatments have been revolutionary in improving outcomes over thelast 30 years, yet cardiovascular disease still exerts a $320B annualburden on the US economy. There is a substantial patient population thatcould benefit from better characterization of risk of major adversecoronary or cerebral events. American Heart Association (AHA),extrapolating ACSD (atherosclerotic cardiovascular disease) risk scoresto the population projects that 9.4% of all adults (age>20) have agreater than 20% risk of adverse events in the next 10 years and 26%have between 7.5% and 20% risk. Applying this to the population yields23 M high risk patients and 57 M moderate risk patients. The 80 M atrisk can be compared to the 30 M U.S. patients are currently on statintherapy in an attempt to avoid new or recurrent events and the 16.5 Mwith a CVD diagnosis. Of those on statins, some will develop occlusivedisease and acute coronary syndrome (ACS). The vast majority of patientsare unaware of their disease progression until onset chest pain. Furtheroutcome and cost improvements in coronary artery disease will flow fromimproved noninvasive diagnostics to identify which patients haveprogressing disease under first line treatments.

Heart health can be significantly impacted by the degradation ofarteries surrounding the heart. A variety of factors (tissuecharacteristics such as angiogenesis, neovascularization, inflammation,calcification, lipid-deposits, necrosis, hemorrhage, rigidity, density,stenosis, dilation, remodeling ratio, ulceration, flow (e.g., of bloodin channel), pressure (e.g., of blood in channel or one tissue pressingagainst another), cell types (e.g., macrophages), cell alignment (e.g.,of smooth muscle cells), or shear stress (e.g., of blood in channel),cap thickness, and/or tortuosity (e.g., entrance and exit angles) by wayof example) may cause these arteries to reduce their effectiveness intransmitting oxygen filled blood to the surrounding tissue (FIG. 35 ).

Functional testing of the coronary arteries, mainlystress-echocardiography and single photon emission computed tomographymyocardial perfusion testing (SPECT MPI), is currently the dominantnoninvasive method for diagnosing obstructive coronary artery disease.Over ten million functional tests are performed each year in the UnitedStates with positive results driving 2.6 M visits to the catheter labfor invasive angiography to confirm the finding of coronary arterydisease.

Another approach to assessing perfusion is to determine thevasculature's ability to transmit oxygen. Specifically, reduced abilitycan be quantified as fractional flow reserve, or FFR. FFR is not adirect measure of ischemia, but rather is a surrogate that measures aratio in pressure drop across the lesion. Changes in luminal diameterrelative to other segments of the same vessel, caused by localvasodilatory impairment at the time of maximal hyperemia, produces amarked hemodynamic effect, leading to abnormal FFR measurement. Duringphysical FFR measurement, the infusion of adenosine decreases downstreamresistance to allow increased flow in the hyperemic state. Physicallymeasuring FFR requires an invasive surgical procedure involving aphysical pressure sensor within the arteries. Because this level ofinvasiveness lends itself to risk and inconvenience there is a demandfor methods that estimate FFR with high accuracy without the need forphysical measurement. The ability to perform this measurementnon-invasively also decreases a noted “treatment bias”: once a patientis in the cath lab, stenting is relatively easy to do so many have notedthat overtreatment occurs whereas if the flow reserve could be assessednon-invasively, improved decisions on whether to stent or not stentcould be possible. Likewise, flow reserve applies to perfusion of braintissues as well (e.g., as related to hyperemia in the brain).

Functional testing has known issues with sensitivity and specificity. Itis estimated that some 30-50% of cardiovascular disease patients aremisclassified and are over-treated or under-treated with significantmonetary and quality of life costs. Functional testing is alsoexpensive, time consuming and of limited use with patients that havepre-obstructive disease. False positives from non-invasive functiontests are a large factor in the overuse of both invasive coronaryangiography and percutaneous coronary interventions in stable patientsthat are a major policy issue in the U.S., Great Britain and China.Studies of the impact of false negatives estimate that of 3.8 M annualMPI tests given to U.S. patients with suspected coronary artery disease(CAD), close to 600,000 will report false negative findings leading to13,700 acute coronary events, many of which would be preventable justthrough introduction of appropriate drug therapies. Another deficiencyof functional testing is temporal in nature: Ischemia is a laggingindicator that follows the anatomical changes brought on by diseaseprogression. Patients at high risk for ACS would also be better servedif future culprit lesions can be detected and reduced with intensivedrug therapy prior to the onset of ischemia.

Coronary computed tomography angiography (CCTA), especially whenutilized in tandem with quantitative analysis software is evolving intoan ideal testing modality to fill this gap in understanding the extentand rate of progression coronary artery disease. Over the last 10 yearsthe CT scanner fleet in most countries has been upgraded tohigher-speed, higher detector count machines capable of excellentspatial resolution without slowing of the heart or extensivebreath-holds. Radiation dose has been greatly lowered, to the pointwhere it is equivalent or lower than SPECT MPI and invasive angiography.

Recent analyses of data from landmark trials like SCOT-HEART, PREDICT,and PROMISE and others have demonstrated the value of detectingnon-obstructive disease, sometimes referred to as high-risk plaque(HRP), vulnerable plaque and/or erosion, using CCTA, by identifyingpatients who are at increased risk for future adverse events. Studydesigns were varied and included nested case-controlled cohortscomparing CCTA registry patients with cardiovascular (CV) events tocontrols with similar risk factors/demographics, comparisons to FFR andmulti-year follow-ups to large “test and treat” studies. The recentfavorable determination from NICE positioning CCTA as a front-linediagnostic is based on a significant reduction in CV events in the CCTAarm of the SCOT-HEART study that was attributed to drug treatmentinitiation or changes on discovery of plaques with CCTA.

An important target patient group are those with stable chest pain andno prior history of CAD with typical or atypical anginal symptoms (basedon SCOT-HEART data), and those with those with non-obstructive disease(<70% stenosis) and in younger patients (e.g., 50-65 years group), basedon the PROMISE findings that suggest that assessment of plaque is mostneeded. Patients with non-obstructive CAD found with high-risk plaqueprofile based on CCTA analysis can be assigned to most appropriate highintensity statin therapy (particularly when a decision on newlipid-lowing therapies that are very expensive such as PCSK9 inhibitors,or anti-inflammatory drugs such as canakinumab are considered), or add anew antiplatelet agent to mitigate the risk of coronary thrombosis,and/or longitudinal follow-up for possible intensification ordowngrading of the therapies. CCTA is an ideal diagnostic tool as it isnoninvasive and requires less radiation than cardiac catheterization.

The pathology literature regarding culprit lesions implicated in fatalheart attacks note that clinically non-obstructive CAD is much morelikely to be home to most high-risk plaque than more occlusive plaqueswhich tend to be more stable. These findings were corroborated by arecent study which noted culprit lesions from ACS patients undergoinginvasive angiography and compared them to precursor plaques in thebaseline CCTA. In one cohort receiving clinically indicated CCTA,patients found to have non-obstructive CAD, 38% of those so tested,still have significant risk of medium to long-term major adverse cardioand/or cerebrovascular events (MACCE). The hazard ratio based on thenumber of diseased segments, independent of obstruction was found to bea significant long-term predictor of MACCE in this group. Onecontributing factor to the predictive value of clinicallynon-obstructive CAD is that these lesions much more likely to be home tomost high-risk plaque than more occlusive plaques which tend to be morestable.

Further demonstration of the potential utility of CCTA in detecting andmanaging obstructive and pre-obstructive atherosclerotic lesions is seenin several recently-published longitudinal studies of statin andanti-inflammatory drug treatment effects, where plaque remodeling tomore stable presentations and plaque regression were observed in thetreatment arms. This corroborates the body of earlier intra-vascularultrasound (IVUS, sometimes with “virtual histology” VH), near-infra redspectroscopy (NIRS), optical coherence tomography (OCT), etc., studiesthat explored disease progression and treatment effect under a varietyof lipid reducing drug protocols. Recent drug trials provide potentialplaque biomarkers to demonstrate efficacy of new medical therapies. TheIntegrated Biomarkers and Imaging Study-4 (IBIS-4) found progression ofcalcification as a potentially protective effect of statins. Otherstudies found reduction in lipid-rich necrotic core (LRNC) under statintreatment. In these studies, clinical variables had poor discriminationof identifying high-risk plaque characteristics when used alone. Thestudies stress the importance of complete characterization andassessment of the entire coronary tree, instead of just the culpritlesion, to allow more accurate risk stratification, which suitablyanalyzed CCTA can do. In a meta-analysis, CCTA had good diagnosticaccuracy to detect coronary plaques compared to IVUS with smalldifferences in assessment of plaque area and volume, percent areastenosis, and slight overestimation of lumen area. Adding rate of changeof lipid rich necrotic core and its distance from the lumen was alsofound to have high prognostic value. Additionally, results from theROMICAT II Trial show that identifying high-risk plaque on CCTA forstable CAD patients with acute chest pain but negative initial ECG andtroponin increases the likelihood of ACS independent of significant CADand clinical risk assessment. Examination by CCTA has been establishedfor evaluation of the coronary atherosclerotic plaques. For patientswhere the necessity of invasive procedures is uncertain, predictingMACCE non-invasively would be beneficial and feasible with CCTA whichgives an overall estimate of disease burden and risk of future events.

The prevalence of carotid artery disease and CAD are closely related.Carotid atherosclerosis has been shown to be an independent predictorfor MACCE, even in patients without pre-existing CAD. Such findingssuggest a common underlying pathogenesis shared in both conditions,which is further supported by the Multi-Ethnic Study of Atherosclerosis(MESA). Atherosclerosis develops progressively through evolution ofarterial wall lesions resulting from the accumulation ofcholesterol-rich lipids and inflammatory response. These changes similar(even if not identical) in the coronary arteries, the carotid arteries,aorta, and peripheral arteries. Certain plaque characteristics such aslarge atheromatous core with lipid-rich content, thin cap, outwardremodeling, infiltration of the plaque with macrophages and lymphocytesand thinning of the media are predisposing to vulnerability and rupture.

e. Non-Invasive Determination of HRP, Erosion and/or FFR:

Non-invasive assessment of the functional significance of stenosis usingCCTA is of clinical and financial interest. The combination of a lesionor vessel's geometry or anatomic structure together with thecharacteristics or composition of the tissue comprising the walls and/orplaque in the walls, collectively referred to as plaque morphology, mayexplain outcomes in lesions with higher or lower risk plaque (HRP),erosion and/or the orthogonal consideration of normal and abnormal flowreserve (FFR). Lesions with a large necrotic core may develop dynamicstenosis due to outward remodeling during plaque formation resulting inmore tissue to stretch, the tissue being stiffer, or the smooth musclelayer being already stretched to the limits of Glagov phenomenon, afterwhich the lesions encroach on the lumen itself. Likewise, inflammatoryinsult and/or oxidative stress could result in local endothelialdysfunction, manifest as impaired vasodilative capacity.

If the tissue making up the plaque is mostly matrix or “fatty streaks”that are not organized into a necrotic core, the plaque dilatessufficiently to keep up with the demand. However, if the plaque has amore substantial necrotic core, it won't be able to dilate. Blood supplywill not be able to keep up with the demand. Plaque morphology increasesaccuracy by evaluating complex factors such as LRNC, calcification,hemorrhage, and ulceration with an objective truth that may be used tovalidate the underlying information, in a manner that other approachescannot due to the lack of an intermediate measurement objectivevalidation.

But that isn't all a plaque can do. Too often, plaques actually rupture,suddenly causing a clot which then may result in infarction of heart orbrain tissue. Plaque morphology also identifies and quantifies thesehigh-risk plaques. For example, plaque morphology may be used todetermine how close the necrotic core is to the lumen: a key determinantof infarction risk. Knowing whether a lesion limits flow under stressdoesn't indicate the risk of rupture or vice versa. Other methods suchas computational fluid dynamics (CFD), without objectively validatedplaque morphology, can simulate flow limitation but not infarction risk.The fundamental advantage of plaque morphology is that its accuracy liesin both the determination of vessel structure and tissuecharacteristics, together allowing determination of phenotype.

Clinical guidelines are increasingly available regarding the optimalmanagement of patients with differing assessments of flow reserve. It isknown that obstructive lesions with high-risk features (large necroticcore and thin cap) portend a maximum likelihood of future events andimportantly, the converse holds true as well.

Without accurate assessments of plaque morphology, approaches todetermine FFR using CFD have been published. But CFD-based flow reserveconsiders the lumen only, or at best, how the luminal surface changes atdifferent parts of the cardiac cycle. Considering only the luminalsurface at best requires processing both systolic and diastolic to getmotion vector (which isn't even done by most available methods), buteven that does not consider what occurs at stress, because theseanalyses are done with computer simulations of what may happen understress rather than measuring actual stress, and not based on the actualcharacteristics that originate vasodilatory capacity but rather just theblood channel. Some methods attempt to simulate forces and applybiomechanical models, but with assumptions rather than validatedmeasurements of wall tissues. Consequently, these methods have noability to anticipate what can occur if stress in fact causes rupturebeyond rudimentary assumptions. Rather, characterizing the tissue solvesthese problems. Wall characteristics, including its effect onvasodilatory capacity of the vessel due to the distensibility of itswalls is considered superior in that lesion makeup determines pliabilityand energy absorption, stable lesions are still over treated, andincomplete assessment of MACCE risk. The advantages of using morphologyto assess FFR include the fact that morphology is a leading indicator,FFR lags, and that presence and degree of HRP, and/or erosion betterinforms treatment for borderline subjects. The importance of solvingaccurate assessment by morphology is strengthened by the studiesincreasingly show that morphology can predict FFR but that FFR does notpredict morphology. That is, effectively assessed morphology has notonly the ability to determine FFR but as well as the likelihood ofdiscontinuous changes in the plaque that move the patient from ischemiato infarction, or HRP.

SUMMARY

Systems and methods are provided herein which utilize a hierarchicalanalytics framework to identify and quantify biologicalproperties/analytes from imaging data and then identify and characterizeone or more medical conditions based on the quantified biologicalproperties/analytes. In some embodiments, the systems and methodsincorporate computerized image analysis and data fusion algorithms withpatient clinical chemistry and blood biomarker data to provide amulti-factorial panel that may be used to distinguish between differentsubtypes of disease. Thus, the systems and methods of the presentdisclosure may advantageously implement biological and clinical insightsin advanced computational models. These models may then interface withsophisticated image processing through rich ontologies that specifyfactors associated with the growing understanding of pathogenesis andtakes the form of rigorous definitions of what is being measured, how itis measured and assessed, and how it is relates to clinically-relevantsubtypes and stages of disease that may be validated.

Human disease exhibits strong phenotypic differences that can beappreciated by applying sophisticated classifiers on extracted featuresthat capture spatial, temporal, and spectral results measurable byimaging but difficult to appreciate unaided. Traditional Computer-AidedDiagnostics make inferences in a single step from image features. Incontrast, the systems and methods of the present disclosure employ ahierarchical inference scheme including intermediary steps ofdetermining spatially-resolved image features and time-resolved kineticsat multiple levels of biologically-objective components of morphology,composition and structure which are subsequently utilized to drawclinical inferences. Advantageously, the hierarchical inference schemeensures the clinical inferences can be understood, validated, andexplained at each level in the hierarchy.

Thus, in example embodiments, systems and methods of the presentdisclosure utilize a hierarchical analytics framework comprised of afirst level of algorithms which measure biological properties capable ofbeing objectively validated against a truth standard independent ofimaging, followed by a second set of algorithms to determine medical orclinical conditions based on the measured biological properties. Thisframework is applicable to a number of distinct biological properties inan “and/or” fashion, i.e., singly or in combination, such asangiogenesis, neovascularization, inflammation, calcification,lipid-deposits, necrosis, hemorrhage, rigidity, density, stenosis,dilation, remodeling ratio, ulceration, flow (e.g., of blood inchannel), pressure (e.g., of blood in channel or one tissue pressingagainst another), cell types (e.g., macrophages), cell alignment (e.g.,of smooth muscle cells), or shear stress (e.g., of blood in channel),cap thickness, and/or tortuosity (e.g., entrance and exit angles) by wayof examples. Measurands for each of these may be measured, such asquantity and/or degree and/or character, of the property. Exampleconditions include perfusion/ischemia (e.g., as limited) (e.g., of brainor heart tissue), perfusion/infarction (as cut off completely) (e.g., ofbrain or heart tissue), oxygenation, metabolism, flow reserve (abilityto perfuse), malignancy, encroachment, and/or risk stratification(whether as probability of event, or time to event (TTE)) e.g., majoradverse cardio- or cerebrovascular events (MACCE). Truth bases mayinclude, for example, biopsy, expert tissue annotations form excisedtissue (e.g., endarterectomy or autopsy), expert phenotype annotationson excised tissue (e.g., endarterectomy or autopsy), physical pressurewire, other imaging modalities, physiological monitoring (e.g., ECG,SaO2, etc.), genomic and/or proteomic and/or metabolomics and/ortranscriptomic assay, and/or clinical outcomes. These properties and/orconditions may be assessed at a given point in time and/or change acrosstime (longitudinal).

In example embodiments, the systems and methods of the subjectapplication, advantageously relate to computer-aided phenotyping (CAP)of disease. CAP is a new and exciting complement to the field ofcomputer-aided diagnosis (CAD). As disclosed herein, CAP may apply ahierarchical inference incorporating computerized image analysis anddata fusion algorithms to patient clinical chemistry and blood biomarkerdata to provide a multi-factorial panel or “profile” of measurementsthat may be used to distinguish between different subtypes of diseasethat would be treated differently. Thus, CAP implements new approachesto robust feature extraction, data integration, and scalablecomputational strategies to implement clinical decision support. Forexample, spatio-temporal texture (SpTeT) method captures relevantstatistical features for characterizing tissue spatially as well askinetically. Spatial features map, for example, to characteristicpatterns of lipid intermixed with extracellular matrix fibers, necrotictissue, and/or inflammatory cells. Kinetic features map, for example, toendothelial permeability, neovascularization, necrosis, and/or collagenbreakdown.

In contrast to current CAD approaches, which make clinical inferences ina single step of machine classification from image features, the systemsand methods of the subject application may advantageously utilize ahierarchical inference scheme may be applied beginning with not onlyspatially-resolved image features but also time-resolved kinetics atmultiple levels of biologically-objective components of morphology andstructural composition in the middle, and then clinical inference at theend. This results in a system that can be understood, validated, andexplained at each level in the hierarchy from low-level image featuresat the bottom to biological and clinical features at the top.

The systems and methods of the present disclosure improve upon bothphenotype classification and outcome prediction. Phenotypeclassification may occur at two levels, individual anatomic locations onthe one hand and more generally described body sites on the other. Theinput data for the former may be 2D data sets, and the input data forthe latter may be 3D data sets. Whereas for phenotype classificationobjective truth may be at either level, for outcome prediction/riskstratification generally occurs at the patient level, but can be morespecific in certain instances (e.g., which side did stroke symptomsmanifest on). The implication here is that the same input data may beused for both purposes, but the models will differ substantially becauseof the level at which the input data will be used as well as the basisof the truth annotations.

While it is possible to perform model building readings vector as inputdata, performance is often limited by the implemented measurands. Thesubject application advantageously utilizes unique measurands (e.g., capthickness, calcium depth, and ulceration) to improve performance. Thus,a readings vector-only approach may be applied where the vector isinclusive of these measurands (e.g., in combination with conventionalmeasurands). The systems and methods of the present disclosure, however,may advantageously utilize a Deep Learning (DL) approach, however, whichcan provide an even richer data set. The systems and methods of thesubject application may also advantageously utilize an unsupervisedlearning application, thereby providing for better scalability acrossdata domains (a very highly desirable feature having in mind the pace atwhich new biomedical data is generated).

In example embodiments, presented herein, Convolutional neural networks(CNNs) may be utilized for building a classifier in an approach that canbe characterized as transfer-learning with fine-tuning approach. CNNstrained on a large compendium of imaging data on a powerfulcomputational platform can be used, with a good success, to classifyimages that have not been annotated in the network training. This isintuitively understandable, as many common classes of features helpidentify images of vastly different objects (i.e. shapes, boundaries,orientation in space, etc.). It is then conceivable that CNNs trained torecognize thousands of different objects using pre-annotated datasets oftens of millions of images would perform basic image recognition tasksmuch better than chance, and would have a comparable performance to CNNstrained from scratch after a relatively minor tweaking of the lastclassification layer, sometimes referred to as the ‘softmax layer. Sincethese models are very large and have been trained on a huge number ofpre-annotated images, they tend ‘to learn’ very distinctive,discriminative imaging features. Thus, the convolutional layers can beused as a feature extractor or the already trained convolutional layerscan be tweaked to suit the problem at hand. The first approach isreferred to as transfer learning and the latter as fine-tuning.

CNNs are excellent at performing many different computer vision tasks.CNNs have a few drawbacks however. Two important drawbacks of importanceto medical systems are 1) the need for vast training and validationdatasets, and 2) intermediate CNN computations are not representative ofany measurable property (sometimes criticized as being a “black box”with undescribed rationale). The approaches disclosed herein mayadvantageously utilize a pipeline consisting of one or more stages whichare separately biologically measurable and capable of independentvalidation, followed by a convolutional neural network starting fromthese rather than only raw imagery. Moreover, certain transforms may beapplied to reduce variation that does not relate to the problem at hand,such as for example unwrapping a donut-shaped vessel cross section tobecome a rectangular representation with a normalized coordinate systemprior to feeding the network. These front-end pipeline stagessimultaneously alleviate both of the two drawbacks of using CNNs formedical imaging.

Generally, the early convolutional layers act as feature extractor ofincreasing specificity, and the fully connected one or two last layersact as classifiers (e.g., “softmax layers”). Schematic representationsof layers sequence and their function in a typical CNN is available frommany sources.

Advantageously, the systems and methods of the present disclosureutilize enriched data sets to enable non-invasive phenotyping of tissuesassayed by radiological data sets. One type of enrichment is topre-process the data to perform tissue classification and use “falsecolor” overlays to provide a data set that can be objective validated(as opposed to only using raw imagery, which does not have thispossibility). Another type of enrichment is to use transformations onthe coordinate system, to accentuate biologically-plausible spatialcontext while removing noise variation to either improve theclassification accuracy, allow for smaller training sets, or both.

In example embodiments, the systems and methods of the subjectapplication may employ a multi-stage pipeline: (i) semantic segmentationto identify and classify regions of interest (e.g., which may berepresentative of quantitative biological analytes) (ii) spatialunwrapping to convert cross-sections of a tubular structure (e.g., avein/artery cross section) into rectangles, and (iii) application of atrained CNN to read the annotated rectangles and identify whichphenotype (e.g., stable or unstable plaque and/or normal or abnormalFFR) it pertains to, and/or predicted time to event (TTE). Note that bytraining and testing a CNN with an unwrapped dataset (with unwrapping)vs a donut dataset (without unwrapping) it can be demonstrated thatunwrapping improves validation accuracy for each particularimplementation. Thus, various implementations imaging of tubularstructures (e.g., plaque phenotyping) or other structures (e.g., lungcancer mass subtyping), or other applications, may similarly, benefitfrom performing similar steps (e.g., semantic segmentation followed byspatial transformations such as unwrapping (prior to applying a CNN).However, it is contemplated that in some alternative embodiments, thatuntransformed datasets (e.g., datasets that are not spatially unwrapped,for example) may be used in determining phenotype (e.g., either inconjunction with or independent of untransformed datasets).

In example embodiments, semantic segmentation and spatial transformationmay involve the following: The image volume may be preprocessedincluding target initialization, normalization, and any other desiredpre-pressing such as deblurring or restoring to form a region ofinterest containing a physiological target that is to be phenotyped.Notably, said region of interest may be a volume composed of crosssections through that volume. The body site may be either automaticallydetermined or is provided explicitly by user. Targets for body sitesthat are tubular in nature may be accompanied with a centerline.Centerlines, when present, can branch. Branches can be labelled eitherautomatically or by user. Note that generalizations on the centerlineconcept may be represented for anatomy that is not tubular but whichbenefit by some structural directionality, e.g., regions of a tumor. Inany case, a centroid may be determined for each cross section in thevolume. For tubular structures this may be the center of the channel,e.g., the lumen of a vessel. For lesions this may be the center of massof the tumor. The (optionally deblurred or restored) image may berepresented in a Cartesian data set where x is used to represent how farfrom centroid, y represents a rotational theta, and z represents thecross section. One such Cartesian set will be formed per branch orregion. When multiple sets are used, a “null” value may be used foroverlapping regions, that is, each physical voxel may be representedonly once across the sets, in such a way as to geometrically fittogether. Each data set can be paired with an additional data set withsub-regions labelled by objectively verifiable tissue composition.Example labels for vascular tissue can be lumen, calcification, LRNC,IPH, etc. Example labels for lesions could be necrotic, neovascularized,etc. These labels can be validated objectively, e.g. by histology.Paired data sets may can used as input to a training step to build aconvolutional neural network. In example embodiments, two levels ofanalysis can be supported, one at an individual cross-section level, anda second at the volume level. Output labels represent phenotype or riskstratification.

Exemplary image pre-processing may include deblurring or restoringusing, for example, a patient-specific point spread determinationalgorithm to mitigate artifacts or image limitations that result fromthe image formation process. These artifacts and image limitations maydecrease the ability to determine characteristics predictive of thephenotype. Deblurring or restoring may be achieved as a result of, forexample, iteratively fitting a physical model of the scanner pointspread function with regularizing assumptions about the true latentdensity of different regions of the image.

In example embodiments, the CNN may be AlexNet, Inception, CaffeNet, orother networks. In some embodiments, refactoring may be done to the CNN,e.g., where a same number and type of layers are used, but the input andoutput dimensions are changed (such as to change the aspect ratio).Example implementations of various example CNNs are provided as opensource on, for example, TensorFlow, and/or in other frameworks,available as open source and/or licensed configurations.

In example embodiments, the dataset may be augmented. For example, insome embodiments, 2D or 3D rotations may be applied to the dataset.Thus, in the case of a untransformed (e.g., donut) dataset, augmentationmay involve, e.g., randomly horizontally flipping the dataset inconjunction with randomly rotating the data set (such as by a randomangle between 0 and 360 degrees). Similarly, in the case of antransformed (e.g., unwrapped) dataset, augmentation may involve e.g.,randomly horizontally flipping in conjunction with a random “scrolling”of the image such as by a random number of pixels in the range from 0 tothe width of the image (where scrolling akin to rotating around theta).

In example embodiments, the dataset may be enriched by using differentcolors to represent different tissue analyte types. These colors may beselected to visually contrast relative to each other, as well asrelative to a non-analyte surface (e.g., normal wall). In someembodiments, a non-analyte surface may be depicted in grey. In exampleembodiments, dataset enrichment may result in ground truth annotation oftissue characteristics (e.g., tissue characteristics that are indicativeof plaque phenotype) as well as provide a spatial context of how suchtissue characteristics present in cross section (e.g., such as takenorthogonal to an axis of the vessel). Such spatial context may include acoordinate system (e.g., based on polar coordinates relative to acentroid of the cross-section) which provides a common basis foranalysis of dataset relative to histological cross sections. Thus,enriched datasets may be advantageously overlaid on top of color-codedpathologist annotations (or vice versa). Advantageously, a histologybased annotated dataset may then be used for training (e.g., training ofthe CNN) in conjunction with or independent of image feature analysis ofa radiological dataset. Notably, a histology based annotated dataset,may improve efficiency in DL approaches since the histology basedannotated dataset uses a relatively simpler false color image in placeof a higher-resolution full image without losing spatial context. Inexample embodiments, coordinate directions may be internally representedusing unit phasors and phasor angle. In some embodiments, the coordinatesystem may be normalized, e.g., by normalizing the radial coordinatewith respect to wall thickness (such as to provide a common basis forcomparing tubular structures/cross-sections of differentdiameters/thicknesses). For example, a normalized radial distance mayhave a value of 0 at an inner (inner wall luminal boundary) and value of1 at an outer boundary (outer wall boundary). Notably, this may beapplied to tubular structures relevant to vascular or otherpathophysiology (e.g., the gastro-intestinal tract).

Advantageously, the enriched datasets of the subject application providefor in vivo non-invasive image-based classification (e.g., where atissue classification scheme can be used to determine phenotypenon-invasively) which is based on a known ground truth. In someembodiments, the known ground truth may be non-radiological (such ashistology or another ex vivo based tissue analysis). Thus, for example,radiology datasets annotated to include ex vivo ground truth data (suchas histology information) may be advantageously used as input data forthe classifier. In some embodiments, a plurality of different knownground truths may be used in conjunction with one another or independentof one another in annotating an enriched dataset.

As noted herein, in some embodiments, an enriched dataset may utilize anormalized coordinate system to avoid non-relevant variation associatedwith, for example, the wall thickness and radial presentation.Furthermore, as noted herein, in example embodiments, a “donut” shapeddataset may be “unwrapped,” e.g., prior to classification training(e.g., using a CNN) and/or prior to running a training classifier on thedataset. Notably, in such embodiments, analyte annotation of thetraining dataset may be prior to transformation, e.g., after unwrappingor a combination of both. For example, in some embodiments, anuntransformed dataset may be annotated (e.g., using ex vivoclassification data such as histology information) and then transformedfor classifier training. In such embodiments, a finer granularity of exvivo based classification may be collapsed to match a lower intendedgranularity for in vivo radiology analysis to not only decreasecomputational complexity but simultaneously address what would otherwisebe open to criticism of being a “black box”.

In some embodiments, colors and/or axes for visualizing the annotatedradiological dataset may be selected to correspond to the samecolors/axes as typically presented in ex vivo ground truth-basedclassifications (e.g., same colors/axes as used in histology). Inexample embodiments, a transformed enriched dataset (e.g., which may benormalized for wall thickness) may be presented where each analyte isvisually represented by a different contrasting color and relative to abackground region (e.g., black or grey) for all non analyte regions.Notably, depending on the embodiment the common background may or maynot be annotated and therefore may or may not visually differentiatebetween non-analyte regions in and out of the vessel wall or betweenbackground features (such as luminal surface irregularity, varying wallthickness, etc.). Thus, in some embodiments, annotated analyte regions(e.g., color coded and normalized for wall thickness) may be visuallydepicted relative to a uniform (e.g., completely black, completely gray,completely white, etc.) background. In other embodiments, annotatedanalyte regions (e.g., color coded but not normalized for wallthickness) may be visually depicted relative to an annotated background(e.g., where different shades (grey, black and/or white) may be used todistinguish between (i) a center lumen region inside the inner lumen ofthe tubular structure, (ii) non-analyte regions inside the wall and/or(iii) a region outside the outer wall. This may enable analysis ofvariations of wall thickness (e.g., due to ulceration or thrombus). Infurther example embodiments, the annotated dataset may include, e.g.,identification of (and visualization of) regions of intra-plaquehemorrhage and/or other morphology aspects. For example, regions ofintra-plaque hemorrhage may be visualized in red, LRNC in yellow, etc.

One specific implementation of the systems and methods of the subjectapplication may be in directing vascular therapy. Classifications may beestablished according to a likely dynamic behavior of a plaque lesion(based on its physical characteristics or specific mechanisms e.g.inflammatory or cholesterol metabolism based) and/or based on aprogression of the disease (e.g., early vs late in its natural history).Such classifications may be used for directing patient treatment. Inexample embodiments, the Stary plaque typing system adopted by the AHAmay be utilized as an underlay with in vivo determined types shown incolor overlays. An example mapping is[‘I’,‘II’,‘III’,‘IV’,‘V’,‘VI’,‘VII’] yielding class_map=[Subclinical,Subclinical, Subclinical, Subclinical, Unstable, Unstable, Stable,Stable]. The systems and methods of the present disclosure are not,however, tied to Stary. as another example, the Virmani system[‘Calcified nodule’, ‘CTO’, ‘FA’, ‘FCP’, ‘Healed Plaque Rupture’, ‘PIT’,‘IPH’, ‘Rupture’, ‘TCFA’, ‘ULC’] has been used with class_map=[Stable,Stable, Stable, Stable, Stable, Stable, Unstable, Unstable, Unstable,Unstable], and other typing systems may yield similarly highperformance. In example embodiments, the systems and methods of thepresent disclosure may merge disparate typing systems, the class map maybe changed, or other variations. For FFR phenotypes, values such asnormal or abnormal may be used, and/or numbers may be used, tofacilitate comparison with physical FFR for example.

Thus, in example embodiments, the systems and methods of the presentdisclosure may provide for phenotype classification of a plaque based onan enriched radiological data set. In particular, the phenotypeclassification(s) may include distinguishing stable plaque from unstableplaque, e.g., where the ground truth basis for the classification isbased on factors such as (i) luminal narrowing (possibly augmented byadditional measures such as tortuosity and/or ulceration), (ii) calciumcontent (possibly augmented by depth, shape, and/or other complexpresentations), (iii) lipid content (possibly augmented by measures ofcap thickness and/or other complex presentations), (iv) anatomicstructure or geometry, and/or (v) IPH or other content. Notably, thisclassification has been demonstrated to have high overall accuracy,sensitivity and specificity as well as a high degree of clinicalrelevance (with potential to change existing standard of care ofpatients who are undergoing catheterization and cardiovascular care).

Another example implementation is lung cancer where the subtypes ofmasses may be determined so as to direct the most likely beneficialtreatment for the patient based on the manifest phenotype. Inparticular, pre-processing and dataset enrichment may be used toseparate out into solid vs. semi-solid (“ground glass”) sub regions,which differ both in degree of malignancy as well as suggestingdiffering optimal treatments.

In further example embodiments, the systems and methods of the presentdisclosure may provide for image pre-processing, image de-noise andnovel geometric representation (e.g., an affine transformation) of CTangiography (CTA) diagnostic images to facilitate and maximizeperformance of Deep Learning Algorithms based on CNNs for developingbest-of-class classifier and a marker of risk of adverse cardiovasculareffects during representation procedures. Thus, as disclosed herein,image deblurring or restoring may be used to identify lesions ofinterest and extract plaque composition quantitatively. Furthermore,transformation of cross-sectional (along the main axis blood vessel)segmented images into, for example, an ‘unwrapped’ rectangular referenceframe which follows the established lumen along the X axis may beapplied to provide a normalized frame to allow DL approaches to bestlearn representative features.

While example embodiments herein utilize 2-D annotated cross-sectionsfor analysis and phenotyping it is noted that the subject application isnot limited to such embodiments. In particular, some embodiments mayutilize enriched 3-D datasets, e.g., instead of or in addition toprocessing 2-D cross-sections separately. Thus, in example embodiments,video interpretation from computer vision may be applied for theclassifier input data set. Note that processing multiple cross-sectionssequentially, as if in a “movie” sequence along a centerline, cangeneralize these methods for tubular structures, e.g., moving up anddown a center-line, and/or other 3D manifestations depending on theaspects most suited to the anatomy.

In further example embodiments false color representations in theenriched data set may have continuous values across pixel or voxellocations. This can be used for, “radiomics” features, with or withoutexplicit validation, or validated tissue types, independently calculatedfor each voxel. Such a set of values may exist in an arbitrary number ofpre-processed overlays and may be fed into the phenotype classifier.Notably, in some embodiments, each pixel/voxel can have values for anynumber of different features (e.g., can be represented in any number ofdifferent overlays for different analytes, sometimes referred to as“multiple occupancy”). Alternatively, each pixel/voxel may only beassigned to one analyte (e.g., assigned to only a single analyteoverlay). Furthermore, in some embodiments, the pixel/voxel value for agiven analyte (e.g., a given analyte) can be based on an all or nothingclassification scheme (e.g., either the pixel is calcium or it isn't).Alternatively, the pixel/voxel value for a given analyte can be arelative value, e.g., a probability score). In some embodiments, therelative values for a pixel/voxel are normalized across a set ofanalytes (e.g., so that the total probability adds up to 100%).

In example embodiments, classification models may be trained in whole orin part by application of multi-scale modeling techniques, such as forexample partial differential equations, e.g., to represent likely cellsignaling pathways or plausible biologically-motivated presentations.

Other alternative embodiments include using change data, for example ascollected from multiple timepoints, rather than (only) data from asingle timepoint. For example, if the amount or nature of a negativecell type increased, it may be said to be a “progressor” phenotype, vs.a “regressor” phenotype for decreases. The regressor might be, forexample, due to response to a drug. Alternatively, if the rate of changefor, say, LRNC is rapid, this may imply a different phenotype, e.g., a“rapid progressor”.

In some embodiments, non-spatial information, such as which are derivedfrom other assays (e.g., lab results), or demographics/risk factors, orother measurements taken from the radiological image, may be fed intothe final layers of the CNN to combine the spatial information withnon-spatial information.

Notably while the systems and methods focus on phenotype classification,similar approaches may be applied with respect to outcome prediction.Such classifications may be based on ground truth historical outcomesassigned to training data sets. For example, life expectancy, quality oflife, treatment efficacy (including comparing different treatmentmethods), and other outcome predictions can be determined using thesystems and methods of the subject application.

Examples of the systems and methods of the subject application arefurther illustrated in the plurality of drawings and the detaileddescription which follows.

In further example embodiments, the systems and methods of the presentdisclosure provide for the determination of fractional flow reserve inmyocardial and/or brain tissue by measurement of plaque morphology.Systems and methods of the present disclosure may use sophisticatedmethods to characterize the vasodilatory capacity of vessels viaobjectively validated determination of tissue type and character whichimpact its distensibility. In particular, plaque morphology may be usedas input to analysis of dynamic behavior of the vasculature from a flowreserve point of view (training the models with flow reserve truthdata). Thus, it is possible to determine the dynamic behavior of thesystem rather than (only) a static description. Stenosis itself is wellknown as being of low predictive power in that it only provides a staticdescription; addition of accurate plaque morphology is necessary, forthe highest accuracy imaging-based assessment of dynamic function. Thepresent disclosure provides systems and methods which determine accurateplaque morphology and then processes such to determine the dynamicfunction.

In example embodiments deep learning is utilized to retain the spatialcontext of tissue characteristics and vessel anatomy (collectivelyreferred to as plaque morphology) at an optimal level of granularity,avoiding excessive non-material variability in the training sets whileretaining that which is needed to exceed other more simplistic use ofmachine learning. Alternative methods by others use only measurements ofvessel structure rather than a more complete processing of tissuecharacteristics. Such methods may capture lesion length, stenosis, andpossibly entrance and exit angles, but they neglect the determinants ofvasodilative capacity. High level assumptions about the flexibility ofthe artery tree as a whole must be made to use these models, but plaquesand other tissue properties cause the distensibility of the coronarytree to be distensible in a heterogeneous way. Different portions of thetree are more or less distensible. Because distensibility is a keyelement in determining FFR, the methods are insufficient. Other methodswhich attempt to do tissue characteristics do so without objectivevalidation as to their accuracy and/or without the data enrichmentmethods needed to retain spatial context optimally for medical imagedeep learning (e.g., transformation such as unwrapping and the validatedfalse color tissue type overlays) necessary to provide the effectivenessof deep learning methods. Some methods try to increase training set sizeby use of synthetic data, but this is ultimately limited to the limiteddata on which the synthetic generation as based and amounts more to adata augmentation scheme rather than a real expansion of the inputtraining set. Additionally, the systems and methods of the presentdisclosure are able to create continuous assessments across vessellengths.

The systems and methods of the present disclosure effectively leverageobjective tissue characterization validated by histology across multiplearterial beds. Of relevance to the example application inatherosclerosis, plaque composition is similar in coronary and carotidarteries, irrespective of its age, and this will largely determinerelative stability, suggesting similar presentation at CCTA as at CTA.Minor differences in the extent of the various plaque features mayinclude a thicker cap and a higher prevalence of intraplaque hemorrhageand calcified nodules in the carotid arteries, however, withoutdifference in the nature of plaque components. In addition, the carotidand coronary arteries have many similarities in the physiology ofvascular tone regulation that has effect on plaque evolution. Myocardialblood perfusion is regulated by the vasodilation of epicardial coronaryarteries in response to a variety of stimuli such as NO, causing dynamicchanges in coronary arterial tone that can lead to multifold changes inblood flow. In a similar fashion, carotid arteries are more than simpleconduits supporting the brain circulation; they demonstrate vasoreactiveproperties in response to stimuli, including shear stress changes.Endothelial shear stress contributes to endothelial health and afavorable vascular wall transcriptomic profile. Clinical studies havedemonstrated that areas of low endothelial shear stress are associatedwith atherosclerosis development and high-risk plaque features.Similarly, in the carotid arteries lower wall shear stress is associatedwith plaque development and localization. (Endothelial shear stress byitself is a useful measurement but not to replace plaque morphology.) Itis important to acknowledge that technical challenges are differentacross beds (e.g. use of gating, vessel size, amount and nature ofmotion)—but these effects are mitigated by scan protocol, which resultin approximate in-plane voxel sizes in the 0.5-0.75 mm range, and thethrough-plane resolution of coronary (the smaller vessels) is actuallybetter than, rather than inferior to, that of carotids (with the voxelsbeing isotropic in coronary and not so in the neck and peripheralextremities).

The present disclosure achieves an effective resolution with routinelyacquired CTA in the same ballpark as IVUS VH, based on solid mathematicsprinciples that respect the Nyquist-Shannon sampling theorem. IVUSimaging has excellent spatial resolution for gross structure (i.e.,lumen) but generally lacks the ability to characterize plaque componentswith high accuracy. Literature estimates of IVUS resolution to be 70-200μm axially and 200-400 μm laterally using typical transducers in the20-40 MHz range. IVUS VH is a method of spectral backscatter analysisthat enables plaque composition analysis (and thus measurements). ForIVUS VH methodology using large (e.g., 480 μm) moving windows in theaxial direction, the relatively large size of this moving window (andthus, the accuracy of the composition analysis) is fundamentally limitedby the bandwidth requirements of the spectral analysis. Where IVUS VHimages are displayed at smaller moving windows, e.g., 250 μm, thislimits the accuracy of this analysis since each IVUS pixel is classifiedinto a discrete category. 64-slice multi-detector CCTA scans have beendescribed to be in the range of 300-400 μm resolution. While thisalready puts CCTA resolution very close to that of IVUS VH, there areadditional factors specific to the present invention analysis toconsider. Thus, rather than discretely classifying CCTA pixels, thesystems and methods of the present disclosure perform an iterativedeblurring or restoring modeling step with sub-voxel precision, e.g.,using a tessellated surface of triangles to represent the true surfaceof lipid core. Lipid core areas are in the range of 6.5-14.3 mm2 in 393patients (corresponding radius of curvature of 1.4-2.1 mm). Using theformula for chord length where the chord spans a single voxeldiagonally, this represents an upper limit on the error of thetessellated surface representation of lipid cores at 44 μm. There areadditional factors associated with the deblurring or restoring analysisthat may cause errors on the order of half a pixel for a total range ofaccuracy of 194-244 μm, generally equivalent to the accuracy of IVUS VHfor measuring cap thickness.

The present disclosure is also innovative in dealing with fundamentallimitations of the application of artificial intelligence and deeplearning to the analysis of atherosclerosis imaging data. Conventionalcompetitive approaches that lack a validated and objective basis such asare fundamentally limited in multiple ways. First, arbitrary thresholdsare used, resulting in an inability to assess accuracy except in a weakform of correlation to other markers that themselves lack objectivevalidation. Second, this lack of objectivity increases the demands forlarge volumes of data on which to circumstantially base correlations.This imposes infeasible demands for manual annotation of theradiological images which are themselves the subject of analysis (thatis, as opposed to being validated from an independent modality).Thirdly, due to the interpretability of the resulting model, thesemodels must be presented to regulatory agencies such as FDA as a “blackbox” which lacks a scientifically rigorous elucidation on mechanisms ofaction that can be tied to traditional biological hypothesis testing.Specifically, whereas CNNs have proven to be excellent at performingmany different computer vision tasks, they have substantial drawbackswhen applied to radiology data sets: 1) the need for vast training andvalidation datasets, and 2) intermediate CNN computations are generallynot representative of any measurable property, which makes regulatoryapproval difficult. To address these challenges, we utilize a pipelineapproach consisting of stages with outputs which are individuallyobjectively capable of validation at the biological level to feed theCNN. The present invention overcomes these drawbacks by using a pipelineconsisting of one or more stages which are biologically measurable (thatis, capable of being objectively validated), followed by smaller scopeconvolutional neural network processing on these validated biologicalproperties to output the desired output conditions not based onsubjective or qualitative “imaging features” but rather. Thesearchitectural capabilities mitigate drawbacks by increasing theefficiency of available training data and make the intermediate stepscapable of being objectively validated. The systems and methods of thepresent disclosure simultaneously alleviate drawbacks of using CNNs formedical imaging by 1) reducing the complexity of the vision task towithin levels that are acceptable when training a CNN with a moderatelysized dataset, and 2) producing intermediate outputs which are bothobjectively validated and easily interpretable by users or regulatingbodies. Intermediate CNN computations are not generally representativeof any measurable property, spatial context is often difficult to obtainin ML approaches using feature extraction, and use of raw data sets thatdo contain spatial context often lack objective ground truth labels forthe extracted features, whether they are processed using traditionalmachine learning or deep learning approaches. Likewise, raw data setsinclude much variation that is “noise” with respect to theclassification problem at hand, which is overcome in computer visionapplications outside of medical by having very large training sets of ascale not generally available in medical applications, especially datasets annotated with ground truth.

The quantitative ability of the systems and methods of the presentdisclosure makes such ideal for analysis of more advanced imagingprotocols (protocols such as early/delayed phase contrast, dual-energy,and multi-spectral techniques are being investigated for tissuecharacterization).

While the systems and methods of the present disclosure have beenparticularly shown and described with reference to example embodimentsthereof, it will be understood by those skilled in the art that variouschanges in form and details may be made therein without departing fromthe scope of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments, as illustrated in the accompanyingdrawings in which like reference characters refer to the same partsthroughout the different views. The drawings are not necessarily toscale, emphasis instead being placed upon illustrating embodiments ofthe present disclosure.

FIG. 1 depicts a schematic of an exemplary system for determining andcharacterizing a medical condition by implementing a hierarchicalanalytics framework, according to the present disclosure.

FIG. 2 outlines a re-sampling based model building approach, accordingto the present disclosure which may be implemented by the systems andmethods described herein.

FIG. 3 depicts a sample patient report, according to the presentdisclosure which may be outputted by the systems and methods describedherein.

FIG. 4 depicts example segmentation levels for a multi-scale vessel wallanalyte map, according to the present disclosure.

FIG. 5 depicts an exemplary pixel-level probability mass function as aset of analyte probability vectors, according to the present disclosure.

FIG. 6 illustrates a technique for computing putative analyte blobs,according to the present disclosure.

FIG. 7 depicts normalized vessel wall coordinates for an exemplaryvessel wall composition model, according to the present disclosure.

FIG. 8 depicts an example margin between plaque removed for a histologyspecimen and the outer vessel wall, according to the present disclosure.

FIG. 9 illustrates some complex vessel topologies which can be accountedfor using the techniques described herein, according to the presentdisclosure.

FIG. 10 depicts representing an exemplary analyte blob with adistribution of normalized vessel wall coordinates, according to thepresent disclosure.

FIG. 11 depicts an exemplary distribution of blog descriptors, accordingto the present disclosure.

FIG. 12 depicts an exemplary model for imaging data correlating betweena hidden ground truth state and an observed state, according to thepresent disclosure.

FIG. 13 depicts a diagram of an example Markov model/Viterbi algorithmfor relating an observed state to a hidden state in an image model,according to the present disclosure.

FIG. 14 depicts an example frequency distribution of total number ofblobs per histological slide for a plurality of histological slides,according to the present disclosure.

FIG. 15 depicts exemplary implantation of a 1D Markov chain, accordingto the present disclosure.

FIG. 16 depicts an example first order Markov chain for a textprobability table, according to the present disclosure.

FIG. 17 depicts conditional dependence of a first pixel based on itsneighboring pixels, according to the present disclosure.

FIG. 18 depicts a further exemplary hierarchical analytics frameworkaccording to the present disclosure.

FIG. 19 depicts an example application of phenotyping purpose indirecting vascular therapy, according to the present disclosure. Thedepicted example uses the Stary plaque typing system adopted by the AHAas an underlay with in vivo determined types shown in color overlays.

FIG. 20 depicts an example application of phenotyping for lung cancer,according to the present disclosure.

FIG. 21 illustrates an exemplary image pre-processing step of deblurringor restoring, according to the present disclosure. The deblurring orrestoring uses a patient-specific point spread determination algorithmto mitigate artifacts or image limitations that result from the imageformation process that may decrease the ability to determinecharacteristics predictive of the phenotype. The deblurred or restored(processed) images depicted (bottom) derive from CT imaging of a plaque(top) and are a result of iteratively fitting a physical model of thescanner point spread function with regularizing assumptions about thetrue latent density of different regions of the image.

FIG. 22 depicts an exemplary application demonstrating an enricheddataset for atherosclerotic plaque phenotype, according to the presentdisclosure.

FIG. 23 illustrates tangential and radial direction variable internallyrepresented using unit phasors and phasor angle (shown coded in greyscale) which exemplifies the use of normalized axes for tubularstructures relevant to the vascular and other pathophysiology associatedwith such structures (e.g., the gastro-intestinal tract), according tothe present disclosure.

FIG. 24 illustrates an exemplary overlay of radiology analysisapplication generated annotations from CTA (outlined regions) on top ofpathologist generated annotations from histology (solid regions),according to the present disclosure.

FIG. 25 demonstrates a further step of data enrichment, specifically,utilizing the normalized coordinate system to avoid non-relevantvariation associated with the wall thickness and radial presentation,according to the present disclosure. Specifically, the “donut” is“unwrapped” while retaining the pathologist annotations.

FIG. 26 represents a further step of data enrichment relevant to plaquephenotyping, according to the present disclosure. Working from theunwrapped formalism, luminal irregularity (for example from ulcerationor thrombus) and local varying wall thickening is represented.

FIG. 27 represents data enriched imaging (in both wrapped and unwrappedforms) including intra-plaque hemorrhage and/or other morphology,according to the present disclosure.

FIG. 28 represents validation results for algorithms trained andvalidated with different variations of data enriched imaging, accordingto the present disclosure.

FIG. 29 provides an example application of phenotyping lung lesions,according to the present disclosure.

FIG. 30 provides an example of biological properties (including e.g.,tissue characteristics, morphology, etc.) for phenotyping lung lesions,according to the present disclosure. Note that in example embodiments,false colors may be represented as continuous values rather thandiscrete values with respect to one or more of these biologicalproperties.

FIG. 31 illustrates a high-level view of one example approach to userinteraction with a computer-aided phenotyping system, according to thepresent disclosure.

FIG. 32 illustrates example system architectures, according to thepresent disclosure.

FIG. 33 illustrates components of example system architectures,according to the present disclosure.

FIG. 34 is a block diagram of example data analysis according to thepresent disclosure. Images are collected of the patient, the raw slicedata is used in a set of algorithms to measure biological propertiesthat may be objectively validated, these are in turn formed as enricheddata sets to feed one of more CNNs, in this example where results areforward and back-propagated using recurrent CNNs to implementconstraints or creates continuous conditions (such as a monotonicallydecreasing fractional flow reserve from proximal to distal throughoutthe vessel tree, or constant HRP value in a focal lesion, or otherconstraints).

FIG. 35 illustrates causality of coronary ischemia and availablediagnostics, according to the present disclosure.

FIG. 36 depicts exemplary 3D segmentations of lumen, vessel wall, andplaque components (LRNC and calcification) for two patients presentingwith chest pain and similar risk factors and degree of stenosis,according to the present disclosure. To the left: 68-year-old man withNSTEMI at follow-up. To the right: 65-year-old man with no event atfollow-up. The systems and methods of the present disclosure correctlypredicted their respective outcomes.

FIG. 37 depicts example histology processing steps for objectivevalidation of tissue composition, according to the present disclosure.

FIG. 38 is a schematic depiction of the multiple cross-sections that maybe processed to provide a dynamic analysis across the entire scannedvessel tree, illustrating the relationships between cross-sections andwhere the processing of one depends on its neighbors, according to thepresent disclosure.

DETAILED DESCRIPTION

Systems and methods for analyzing pathologies utilizing quantitativeimaging are presented herein. Advantageously, the systems and methods ofthe present disclosure utilize a hierarchical analytics framework thatidentifies and quantify biological properties/analytes from imaging dataand then identifies and characterizes one or more pathologies based onthe quantified biological properties/analytes. This hierarchicalapproach of using imaging to examine underlying biology as anintermediary to assessing pathology provides many analytic andprocessing advantages over systems and methods that are configured todirectly determine and characterize pathology from raw imaging datawithout the validation steps and/or without advantageous processingdescribed herein.

One advantage, for example, is the ability to utilize training sets fromnon-radiological sources, e.g., from tissue sample sources such ashistological information, in conjunction with or independent of trainingsets for radiological sources, to correlate radiological imagingfeatures to biological properties/analytes to pathologies. For example,in some embodiments, histology information may be used in trainingalgorithms for identifying and characterizing one or more pathologiesbased on quantified biological properties/analytes. More specifically,biological properties/analytes which are identifiable/quantifiable innon-radiological data (such as in an invasively obtained histology dataset or obtainable via gene expression profiling) may also be identifiedand quantified in radiological data (which is advantageouslynon-invasive). These biological properties/analytes may then becorrelated to clinical findings on pathology using information the fromnon-radiological sources, for example, utilizing histologicalinformation, gene expression profiling, or other clinically rich datasets. This set of clinically correlated data may then serve as atraining set or part of a training set for determining/tuning (e.g.,utilizing machine learning) algorithms correlating biologicalproperties/analytes to pathologies with a known relationship to clinicaloutcome. These algorithms correlating biological properties/analytes topathologies derived utilizing non-radiological source training sets maythen be applied in evaluating biological properties/analytes derivedfrom radiological data. Thus, the systems and methods of the presentdisclosure may advantageously enable utilizing radiological imaging(which may advantageously be cost-effective and non-invasive) to providesurrogate measures for predicting clinical outcome or guiding treatment.

Notably, in some instances training data for non-radiological sources(such as histology information) may be more accurate/reliable thantraining data for radiological sources. Moreover, in some embodiments,training data from non-radiological sources may be used to augmenttraining data from radiological sources. Thus, since better data in islikely to yield better data out, the hierarchical analytics frameworkdisclosed advantageously improves the trainability and resultingreliability of the algorithms disclosed herein. As noted above, one keyadvantage is that, once trained the systems and methods of the presentdisclosure may enable deriving comparable clinical information toexisting histological and other non-radiological diagnostic-type testingwithout the need not undergo invasive and/or costly procedures.

Alternatively, in some embodiments, training sets for non-radiologicalsources (such as non-radiological imaging sources, e.g., histologicalsources, and/or non-imaging sources) may be utilized in conjunction withor independent of training sets for radiological sources, e.g., incorrelating image features to biological properties/analytes. Forexample, in some embodiments one or more biological models may beextrapolated and fitted to correlate radiological and non-radiologicaldata. For example, histology information may be correlated withradiological information based on an underlying biological model. Thiscorrelation may advantageously enable training recognition of biologicalproperties/analytes in radiological data utilizing non-radiological,e.g., histological information.

In some embodiments, data drawn from complementary modalities may beused, e.g., in correlating image features to biologicalproperties/analytes from blood panels, physical FFR, and/or othersources of data.

In example embodiments one or more biological models may be extrapolatedand fitted utilizing imaging data drawn from one imaging modality eithercorrelated with and/or fused with another imaging modality ornon-imaging source such as bloodwork. These biological models mayadvantageously correlate across and between imaging and non-imaging datasets based on the biological models. Thus, these biological models mayenable the hierarchical analytics framework to utilize data from oneimaging modality with another imaging modality or with a non-imagingsource in identifying/quantifying one or more biologicalproperties/analytes or identifying/characterizing one or more medicalconditions.

Another advantage to the hierarchical analytics framework disclosedherein, is the ability to incorporate data from multiple same ordifferent type data sources into the process of identifying andcharacterizing pathology based on imaging data. For example, in someembodiments, one or more non-imaging data sources may be used inconjunction with one or more imaging data sources in identifying andquantifying a set of biological properties/analytes. Thus, inparticular, the set of biological properties/analytes may include one ormore biological properties/analytes identified and/or quantified basedon one or more imaging data sources, one or more biologicalproperties/analytes identified and/or quantified based on one or morenon-imaging data sources, and/or one or more biologicalproperties/analytes identified and/or quantified based on a combinationof imaging and non-imaging data sources (note that, for the purposes ofthe quantitative imaging systems and methods of the present disclosurethe set of biological properties/analytes may generally include at leastone or more biological properties/analytes identified and/or quantifiedbased at least in part on an imaging data). The ability to augmentinformation from an imaging data source with information from otherimaging and/or non-imaging data sources in identifying and quantifying aset of biological properties/analytes adds to the robustness of thesystems and methods presented herein and enables utilization of any andall relevant information in identifying and characterizing pathology.

Yet another advantage of the hierarchical analytics framework involvesthe ability to adjust/fine-tune data at each level, e.g., prior orsubsequent to utilizing that data to assess the subsequent level (notethat in some embodiments this may be an iterative process). For example,in some embodiments, information related to a set of identified andquantified biological properties/analytes may be adjusted in an aposteriori manner (e.g., after an initial identification and/orquantification thereof). Similarly, in some embodiments, informationrelated to a set of identified and characterized pathologies may beadjusted in an a posteriori manner (e.g., after an initialidentification and/or characterization thereof). These adjustments maybe automatic or user based and may objective or subjective. The abilityto adjust/fine-tune data at each level may advantageously improve dataaccountability and reliability.

In example embodiments, adjustments may be based on contextualinformation, which may be used to update one or more probabilitiesimpacting a determination or quantification of a biologicalproperty/analyte. In example embodiments, contextual information foradjusting information related to a set of identified and quantifiedbiological properties/analytes in an a posteriori manner may includepatient demographics, correlations between biologicalproperties/analytes or correlations between identified/characterizedpathologies and biological properties/analytes. For example, in someinstances the biological properties/analytes may be related in the sensethat the identification/quantification of a first biologicalproperty/analyte may impact a probability relating theidentification/quantification of a second biological property/analyte.In other instances, identification/characterization of a firstpathology, e.g., based on an initial set of identified/quantifiedbiological properties/analytes may impact a probability relating to theidentification/quantification of a biological property/analyte in theinitial set or even a biological property/analyte that wasn't in thefirst set. In further instances, pathologies may be related, e.g.,wherein identification/characterization of a first pathology may impacta probability relating the identification/characterization of a firstpathology. As noted above, information related to identification andquantification of biological properties/analytes and/or informationrelated to the identification and characterization of pathologies may beupdated in an iterative manner, e.g., until data convergence orthresholds/benchmarks are achieved or for a selected number of cycles.

A further advantage of the hierarchical analytics framework involves theability to provide a user, e.g., a physician, with information relatingboth to a pathology as well as the underlying biology. This addedcontext may facilitate clinical diagnosis/evaluation as well asassessing/determining next steps, e.g., therapeutic/treatment options orfurther diagnostics. For example, the systems and methods may beconfigured to determine which biological parameters/analytes relevant tothe identification/quantification of one or more pathologies are mostindeterminate/have the highest degree of uncertainty (e.g., by reason oflack of data or conflicting data). In such instances, specific furtherdiagnostics may be recommended. The added context of providing a userwith information relating both to a pathology as well as the underlyingbiology may further help the user evaluate/error check various theclinical conclusions and recommendations reached by the analytics.

A hierarchical analytics framework, as used herein, refers to ananalytic framework wherein a one or more intermediary sets of datapoints are utilized as an intermediary processing layer or anintermediary transformation between initial set of data points and anend set of data points. This is similar to the concept of deep learningor hierarchical learning wherein algorithms are used to model higherlevel abstractions using multiple processing layers or otherwiseutilizing multiple transformations such as multiple non-lineartransformations. In general, the hierarchical analytics framework of thesystems and methods of the present disclosure includes data pointsrelating to biological properties/analytes as an intermediary processinglayer or intermediary transformation between imaging data points andpathology data points, in example, embodiments, multiple processinglayers or multiple transformation (e.g., as embodied by multiple levelsof data points) may be included for determining each of imaginginformation, underlying biological information and pathologyinformation. While example hierarchical analytic framework structuresare introduced herein (e.g., with specific processing layers, transformsand datapoints), the systems and methods of the present disclosure arenot limited to such implementations. Rather, any number of differenttypes of analytic framework structures may be utilized without departingfrom the scope and spirit of the present disclosure.

In example embodiments, the hierarchical analytics frameworks of thesubject application may be conceptualized as including a logical datalayer as an intermediary between an empirical data layer (includingimaging data) and a results layer (including pathology information).Whereas the empirical data layer represents directly sourced data thelogical data layer advantageously adds a degree of logic and reasoningwhich distills this raw data into a set of useful analytes for theresults layer in question. Thus, for example, empirical information fromdiagnostics such as raw imaging information may be advantageouslydistilled down to a logical information relating to a particular set ofbiological features which is relevant for assessing a selected pathologyor group of pathologies (for example, pathologies related to an imagedregion of the patient's body). In this way the biologicalfeatures/analytes of the subject application can also be thought of aspathology symptoms/indicators.

The biological features/analytes of the subject application may at timesbe referred to herein a biomarkers. While the term “biological” orprefix “bio” is used in characterizing biological features or biomarkersthis in only intended to signify that the features or markers have adegree of relevance with respect to the patient's body. For example,biological features may be anatomical, morphological, compositional,functional, chemical, biochemical, physiological, histological, geneticor any number of other types of features related to the patient's body.Example, biological features utilized by specific implementations of thesystems and methods of the present disclosure (e.g., as relating toparticular anatomical regions of a patient such as the vascular system,the respiratory system, organs such as the lungs, heart or kidneys, orother anatomical regions) are disclosed herein.

While example systems and methods of the present disclosure may begeared toward detecting, characterizing and treatingpathologies/diseases, the application of the systems and methods of thepresent disclosure are not limited to pathologies/diseases but rathermay more generally applicable with respect to any clinically relevantmedical conditions of a patient including, e.g., syndromes, disorders,traumas, allergic reactions, etc.

In exemplary embodiments, the systems and methods of the presentdisclosure relate to Computer-Aided Phenotyping, e.g., by usingknowledge about biology to analyze medical images to measure thedifferences between disease types that have been determined throughresearch to indicate phenotypes which in turn predict outcomes. Thus, insome embodiments, characterizing pathologies may include determiningphenotypes for the pathologies which may in turn determine a predictiveoutcome.

With initial reference to FIG. 1 , a schematic of an exemplary system100 is depicted. There are three basic functionalities which may beprovided by the system 100 as represented by the trainer module 110, theanalyzer module 120 and the cohort tool module 130. As depicted, theanalyzer module 120 advantageously implements a hierarchical analyticsframework which first identifies and quantifies biologicalproperties/analytes 130 utilizing a combination of (i) imaging features122 from one or more acquired images 121A of a patient 50 and (ii)non-imaging input data 121B for a patient 50 and then identifies andcharacterizes one or more pathologies (e.g., prognostic phenotypes) 124based on the quantified biological properties/analytes 123.Advantageously, the analyzer module 120 may operate independent ofground truth or validation references by implementing one or morepre-trained, e.g., machine learned algorithms for drawing itsinferences.

In example embodiments, the analyzer may include algorithms forcalculating imaging features 122 from the acquired images 121A of thepatient 50. Advantageously, some of the image features 122 may becomputed on a per-voxel basis while others may be computed on aregion-of-interest basis. Example non-imaging inputs 121B which may beutilized along with acquired images 121A may include data fromlaboratory systems, patient-reported symptoms, or patient history.

As noted above, the image features 122 and non-imaging inputs may beutilized by the analyzer module 120 to calculate the biologicalproperties/analytes 123. Notably, the biological properties/analytes aretypically quantitative, objective properties (e.g., objectivelyverifiable rather than being stated as impression or appearances) thatmay represent e.g., a presence and degree of a marker (such as achemical substance) or other measurements such as structure, size, oranatomic characteristics of region of interest. In example embodiments,the quantified biological properties/analytes 123 may be displayed orexported for direct consumption by the user, e.g., by a clinician, inaddition to or independent of further processing by the analyzer module.

In example embodiments, one or more of the quantified biologicalproperties/analytes 123 may be used as inputs for determining phenotype.Phenotypes are typically defined in a disease-specific mannerindependent of imaging, often being drawn from ex vivopathophysiological samples for which there is documented relationship tooutcome expected. In example embodiments, the analyzer module 120 mayalso provide predicted outcomes 125 for determined phenotypes.

It should be appreciated that example implementations of the analyzermodule 120 are further described herein with respect to specificembodiments which follow the general description of the system 100. Inparticular, specific imaging features, biological properties/analytesand pathologies/phenotypes are described with respect to specificmedical applications such as with respect to the vascular system or withrespect to the respiratory system.

With reference still to FIG. 1 , the cohort tool module 130 enablesdefining a cohort of patients for group analyses thereof, e.g., based ona selected set of criteria related to the cohort study in question. Anexample cohort analysis may be for a group of patients enrolled in aclinical trial, e.g., with the patient's further being grouped based onone or more arms of the trial for example a treatment vs. control arm.Another type of cohort analysis may be for a set of subjects for whichground truth or references exist, and this type of cohort may be furtherdecomposed into a training set or “development” set and a test or“holdout” set. Development sets may be supported so as to train 112 thealgorithms and models within analyzer module 120, and holdout sets maybe supported so as to evaluate/validate 113 the performance of thealgorithms or models within analyzer module 120.

With continued reference to FIG. 1 , the trainer module 110 may beutilized to train 112 the algorithms and models within analyzer module120. In particular, the trainer module 110, may rely on ground truth 111and/or reference annotations 114 so as to derive weights or models,e.g., according to established machine learning paradigms or byinforming algorithm developers. In example embodiments, classificationand regression models are employed which may be highly adaptable, e.g.,capable of uncovering complex relationships among the predictors and theresponse. However, their ability to adapt to the underlying structurewithin the existing data can enable the models to find patterns that arenot reproducible for another sample of subjects. Adapting toirreproducible structures within the existing data is commonly known asmodel over-fitting. To avoid building an over-fit model, a systematicapproach may be applied that prevents a model from finding spuriousstructure and enable the end-user to have confidence that the finalmodel will predict new samples with a similar degree of accuracy on theset of data for which the model was evaluated.

Successive training sets may be utilized to determine optimal tuningparameter(s), and a test set may be utilized to estimate an algorithm'sor model's predictive performance. Training sets may be used fortraining each of the classifiers via randomized cross-validation.Datasets may be repeatedly split into training and testing sets and maybe used to determine classification performance and model parameters.The splitting of the datasets into training and test sets occurs using astratified or maximum dissimilarity approaches. In example embodiments are-sampling approach (e.g. bootstrapping) may be utilized within thetraining set in order to obtain confidence intervals for (i) the optimalparameter estimate values, and (ii) the predictive performance of themodels.

FIG. 2 outlines a re-sampling based model building approach 200 whichmay be utilized by the systems and methods of the present disclosure.First, at step 210, a tuning parameter set may be defined. Next, at step220, for each tuning parameter set data is resampled the model is fittedand hold-out samples are predicted. At step 230, Resampling estimatesare combined into a performance profile. Next, at step 240, final tuningparameters may be determined. Finally, at step 250, the entire trainingset is re-fitted with the final tuning parameters. After each model hasbeen tuned from the training set, each may be evaluated for predictiveperformance on the test set. Test set evaluation occur once for eachmodel to ensure that the model building process does not over-fit thetest set. For each model that is constructed, the optimal tuningparameter estimates, the re-sampled training set performance, as well asthe test set performance may be reported. The values of the modelparameters over randomized splits are then be compared to evaluate modelstability and robustness to training data.

According to the systems and methods of the present disclosure, a numberof models may be tuned for each of the biological properties/analytes(e.g., tissue types) represented in ground truth maps. Model responsesmay include, for example, covariance-based techniques, non-covariancebased techniques, and tree based models. Depending on theirconstruction, endpoints may have continuous and categorical responses;some of the techniques in the above categories are used for bothcategorical and continuous responses, while others are specific toeither categorical or continuous responses. Optimal tuning parameterestimates, the re-sampled training set performance, as well as the testset performance may be reported for each model.

TABLE 1 Delineate Field Register multiple data streams across a fieldSegment organs, vessels, lesions, and other application-specific objectsReformat anatomy for specific analyses Delineate Target Registermultiple data streams at a locale Fine-grained segmentation Measure sizeand/or other relevant anatomic structure Extract whole-target featuresDelineate Sub-target Split target into sub-targets according to regionsapplication Sub-target specific calculations Delineate (Re-) SegmentComponent Components Calculate Readings Visualize Probability MapDetermine Disease Determine Phenotype Severity Predict Outcome CompareMultiple (Optional) Compare Multiple Timepoints Timepoints Assessmulti-focal Aggregate across target lesions over a wide disease scanfield. Generate Patient Generate Patient Report Report

Table 1, above, provides a summary of some of the examplefunctionalities of the analyzer module 120 of system 100. Namely, theanalyzer module 120 may be configured to delineate fields, for example,to register multiple data streams across a field; to segment organs,vessels, lesions and other application-specific objects; and/or toreformat/reconfigure anatomy for specific analyses. The analyzer module120 may further be configured for delineating a target, for example, alesion, in a delineated field. Delineating a target may, for example,include registering multiple data streams at a locale; conductingfine-grained segmentation; measuring size and/or other characteristicsof relevant anatomic structures; and/or extracting whole-target features(e.g., biological properties/analytes characteristic of the entiretarget region). In some embodiments, one or more sub-target regions mayalso be delineated, for example, a target region may be split intosub-targets according to a particular application with sub-targetspecific calculations (e.g., biological properties/analytescharacteristic of a sub-target region). The analyzer module 120 may alsodelineate components or relevant features (such as composition), forexample, in a particular field, target or sub-target region. This mayinclude segmenting or re-segmenting the components/features, calculatingvalues for the segmented components/features (e.g., biologicalproperties/analytes characteristic of the component/feature) andassigning a probability map to the readings. Next pathologies may bedetermined, based on the biological quantified properties/analytes, andcharacterized, e.g., by determining phenotype and/or predictive outcomesfor the pathologies. In some embodiments, the analyzer module 120 may beconfigured to compare data across multiple timepoints, e.g., one or moreof the biological components/analytes may involve a time-basedquantification. In further embodiments, a wide scan field may beutilized to assess multi-focal pathologies, e.g., based on aggregatequantifications of biological properties/analytes across a plurality oftargets in the delineated field. Finally, based on the forgoinganalytics, the analyzer module 120 may be configured to generate apatient report.

A sample patient report 300 is depicted in FIG. 3 . As shown, the samplepatient report 300 may include quantifications of biologicalparameters/analytes such as relating to structure 310 and composition320 as well as data from non-imaging sources such as hemodynamics 330.The sample patient report may further include visualizations 340, e.g.,2D and/or 3D visualizations of imaging data as well as combinedvisualizations of non-imaging data such as hemodynamic data overlaidonto imaging data. Various analytics 350 may be displayed for assessingthe biological parameters/analytes including, e.g., a visualization ofone or more model(s) (e.g., a decision tree model) fordetermining/characterizing pathology. Patient background and identifyinginformation may further be included. Thus, the analyzer module 120 ofsystem 100 may advantageously provide a user, e.g., a clinician withcomprehensive feedback for assessing the patient.

Advantageously the systems and methods of the present disclosure may beadapted for specific applications. Example vascular and lungapplications are described in greater detail in the sections whichfollow (although it will be appreciated that the specific applicationdescribed have general implications and interoperability with respect tonumerous other applications). Table 2 provides an overview of vascularand lung related applications utilizing a hierarchical analyticsframework as described herein.

TABLE 2 Vascular Application Lung Application Modality CT or MR CTIndication Asymptomatic CAS Lung Cancer Screening Cryptogenic strokeDrug therapy response NSTEMI, CABG Patency assessment EvaluationCompanion-diagnostic for Companion-diagnostic for expensive or targeteddrugs expensive or targeted drugs Diseases Peripheral and coronary Lungcancer first, then other artery vasculopathy pulmonary diseaseBiological Structure Size, Shape/Margin Properties Composition Solidity,Heterogeneity Hemodynamics Invasive Potential Gene Expression CorrelatesGene Expression Correlates Extension Ultrasound and/or multi- PET and/ormulti-energy CT energy CT

The following sections provide specific examples of quantitativebiological properties/analytes that may be utilized by the systems andmethods of the present disclosure with respect to vascular applications:

Anatomic Structure: Vessel structural measurements, specifically thosethat lead to the determination of % stenosis, have long been and remainthe single most used measurements in patient care. These were initiallylimited to inner lumen measurements, rather than wall measurementsinvolving both the inner and outer surfaces of the vessel wall. However,all of the major non-invasive modalities, unlike X-ray angiography, canresolve the vessel wall and with this come expanded measurements thatmay be achieved. The category is broad and the measurements are ofobjects of varying sizes, so generalizations should be made with care. Aprimary consideration is the limit of spatial sampling or resolution.The minimally detectable changes in wall thickness may, however, belower than the spatial sampling by taking advantage of subtle variationsin intensity levels due to partial volume effect. Additionally, statedresolutions generally refer to grid size and field of view ofpost-acquisition reconstructions rather than the actual resolving powerof the imaging protocol, which determines the minimum feature size thatcan be resolved. Likewise, in-plane vs. through-plane resolutions may ormay not be the same and not only the size of a given feature but as wellits proportions and shape will drive the measurement accuracy. Last butnot least, in some cases categorical conclusions are drawn from applyingthresholds to the measurements, which may then be interpreted accordingto signal detection theory with the ability to optimize the trade-offbetween sensitivity and specificity, terms that don't otherwise refer tomeasurements in the normal sense.

Tissue Characteristics: The quantitative assessment of the individualconstituent components of the atherosclerotic plaques, including forexample lipid rich necrotic core (LRNC), fibrosis, intraplaquehemorrhage (IPH), permeability, and calcification, can provide crucialinformation concerning the relative structural integrity of the plaquethat could aid the physician's decisions on course of medical orsurgical therapy. From the imaging technology point of view, the abilityto do this lies less with spatial resolution as with contrast resolutionand tissue discrimination made possible by differing tissues respondingto incident energy differently so as to produce a differing receivesignal. Each imaging modality does this to some extent; terms inultrasound such as “echolucency”, the CT number in Hounsfield Units, anddifferentiated MR intensities as a function of various sequences such as(but not limited to) T1, T2 and T2*.

Dynamic tissue behavior (e.g., Permeability): In addition tomorphological features of the vessel wall/plaque, there is increasingrecognition that dynamic features are valuable quantitative indicatorsof vessel pathology. Dynamic sequences where the acquisition is taken atmultiple closely-spaced times (known as phases) expand the repertoirebeyond spatially-resolved values t include temporally-resolved valueswhich may be used for compartment modeling or other techniques todetermine the tissues' dynamic response to stimulus (such as but notlimited to wash-in and wash-out of contrast agent). Through the use ofdynamic contrast enhanced imaging with ultrasound or MR in the carotidarteries or delayed contrast enhancement in the coronary arteries,sensitive assessments of the relative permeability (e.g., Ktrans and Vpparameters from kinetic analysis) of the microvascular networks ofneoangiogenesis within the plaques of interest can be determined. Inaddition, these dynamic series can also aid in the differentiationbetween increased vascular permeability versus intraplaque hemorrhage.

Hemodynamics: The basic hemodynamic parameters of the circulation have adirect effect on the vasculopathy. Blood pressures, blood flow velocity,and vessel wall shear stress may be measured by techniques ranging fromvery simple oscillometry to sophisticated imaging analysis. Using commonprinciples of fluid dynamics, calculations of vessel wall shear stresscan be ascertained for different regions of the wall. In similar fashionMRI, with or without the combination of US, has been used to calculatethe wall shear stress (WSS) and correlate the results with structuralchanges in the vessel of interest. In addition, the effects ofantihypertensive drugs on hemodynamics have been followed for short andlong-term studies.

Thus, in example embodiments, key aspects of applying the systems andmethods of the present disclosure in a vascular setting may includeevaluating plaque structure and plaque composition. Evaluating plaquestructure may advantageously include, e.g., lumen measurements (whichimproves stenosis measurement by providing area rather than onlydiameter measures) as well as wall measurements (e.g., wall thicknessand vascular remodeling). Evaluating plaque composition mayadvantageously involve quantification of tissue characteristics (e.g.,lipid core, fibrosis, calcification, permeability, etc.) rather thanjust “soft” or “hard” designations as typically found in the prior art.Tables 3 and 4, below, describe example structural calculations andtissue characteristic calculations, respectively which may be utilizedby the vascular applications of the systems and methods of the presentdisclosure.

TABLE 3 Structural calculations of vessel anatomy supported by vascularapplications of the systems and methods disclosed herein. MeasurandDescription Type and Units Remodeling Calculated as the ratio of vesselExpressed with value Ratio area with plaque to reference less than 1 forinward vessel wall area without plaque remodeling and greater than 1 foroutward remodeling % Stenosis Calculated as the (1 − ratio of Expressedas minimum lumen with plaque to percentage > 0% reference lumen withoutplaque) × 100 both by area and by diameter % Dilation Calculated as the(ratio of Expressed as maximum lumen with plaque to percentage > 0%reference lumen without plaque − 1) × 100 both by area and diameter WallCalculated by measuring the Expressed in units of Thickness largestthickness of wall mm

TABLE 4 Calculations of tissue characteristics supported by vascularapplications of the systems and methods disclosed herein MeasurandDescription Type and Units Lipid Core The pathologic retention oflipids, Burden in mm² particularly lipoproteins, by intimal/medial bycross section cells leading to progressive cell loss, cell and mm³ bydeath, degeneration, and necrosis. It is a target and mixture of lipid,cellular debris, blood and vessel water in various concentrations.Fibrosis The pathologic and sometimes physiologic Burden in mm²defensive production of fibrous tissue by by cross section fibroblastsand activated smooth muscle and mm³ by cells. target and vesselCalcification The physiologic defensive biological Agatston scoreprocess of attempting to stabilize plaque, and burden in which has amechanism akin to bone mm² by cross formation. section and mm³ by targetand vessel Hemorrhage A pathologic component that may Burden in mm²contribute to the vulnerability of a plaque. by cross section Its roleis not fully understood, but it is and mm³ by believed to be a drivingforce in plaque target and progression through lipid accumulation vesselfrom red blood cells. Permeability Described as endothelial and intimalBurden in mm² permeability due to neovascularization, by cross sectionnecrosis, collagen breakdown, and and mm³ by inflammation target andvessel Thrombosis Local coagulation or clotting of the blood Degree in apart of the circulatory system. Ulceration Disintegration and necrosisof epithelial Burden in mm² tissue by cross section and mm³ by targetand vessel

Example systems relating to evaluating the vascular system mayadvantageously include/employ algorithms for evaluating vascularstructure. Thus, the systems may employ, e.g., a target/vesselsegment/cross-section model for segmenting the underlying structure ofan imaged vessel. Advantageously a fast-marching competition filter maybe applied to separate vessel segments. The systems may further beconfigured to handle vessel bifurcations. Image registrations may beapplied utilizing Mattes mutual information (MR) or mean square error(CT) metric, rigid versor transform, LBFGSB optimizer, or the like. Asnoted herein, vessel segmentation may advantageously include lumensegmentation. An initial lumen segmentation may utilize a confidenceconnected filter (e.g., carotid, vertebral, femoral, etc.) todistinguish the lumen. Lumen segmentation may utilize MR imaging (suchas a combination of normalized, e.g., inverted for dark contrast,images) or CT imaging (such as use of registered pre-contrast,post-contrast CT and 2D Gaussian distributions) to define a vessel-nessfunction. Various connected components may be analyzed and thresholdingmay be applied. Vessel segmentation may further entail outer wallsegmentation (e.g., utilizing a minimum curvature (k2) flow to accountfor lumen irregularities). In some embodiments, an edge potential map iscalculated as outward-downward gradients in both contrast andnon-contrast. In example embodiments, outer wall segmentation mayutilize cumulative distribution functions (incorporating priordistributions of wall thickness, e.g., from 1-2 adjoining levels) in aspeed function to allow for median thickness in the absence of any otheredge information. In example embodiments, Feret diameters may beemployed for vessel characterization. In further embodiments, wallthickness may be calculated as the sum of the distance to lumen plus thedistance to the outer wall. In further embodiments, lumen and/or wallsegmentations may be done using semantic segmentation using, forexample, CNNs.

Example systems relating to evaluating the vascular system may furtheradvantageously analyze vascular composition. For example, in someembodiments, composition may be determined based on image intensity andother image features. In some embodiments, the lumen shape may beutilized, e.g., as relating to determining thrombosis. Advantageously,an analyte blob model may be employed for better analyzing compositionof particular sub-regions of the vessel. We define an analyte blob to bea spatially contiguous region, in 2D, 3D, or 4D images, of one class ofbiological analyte. The blob model may utilize an anatomically alignedcoordinate system using isocontours, e.g., in normalized radial distancefrom the lumen surface to the adventitial surface of the vessel wall.The model may advantageously identify one or more blobs and analyze eachblobs location e.g., with respect to the overall vessel structure aswell as relative to other blobs. In example embodiments, a hybridBayesian/Markovian network may be utilized to model a relative locationof a blob. The model may advantageously account for the observed imageintensity at a pixel or voxel being influenced by a local neighborhoodof hidden analyte category nodes thereby accounting for partial volumeand scanner point spread function (PSF). The model may further allow fordynamically delineating analyte blob boundaries from analyte probabilitymaps during inference by the analyzer module. This is a key distinctionfrom typical machine vision approaches, such as with superpixelapproaches, that pre-compute small regions to be analyzed but are unableto dynamically adjust these regions. An iterative inference proceduremay be applied that utilizes uses the current estimate of both analyteprobability and blob boundaries. In some embodiments parametric modelingassumptions or kernel density estimation methods may be used to enableprobability density estimates between the sparse data used to train themodel.

Introduced herein is a novel model for classification of composition ofvascular plaque components that removes the requirements forhistology-to-radiology registration. This model still utilizesexpert-annotated histology as a reference standard but the training ofthe model does not require registration to radiological imaging. Themulti-scale model computes the statistics of each contiguous region of agiven analyte type, which may be referred to as a ‘blob’. Within across-section through the vessel, the wall is defined by two boundaries,the inner boundary with the lumen and the outer boundary of the vesselwall, creating a donut shape in cross section. Within the donut shapedwall region, there are a discrete number of blobs (different than thedefault background class of normal wall tissue which is not consideredto be a blob). The number of blobs is modeled as a discrete randomvariable. Then, each blob is assigned a label of analyte type andvarious shape descriptors are computed. Additionally, blobs areconsidered pairwise. Finally, within each blob, each pixel can produce aradiological imaging intensity value, which are modeled as independentand identically distributed (i.i.d.) samples that come from acontinuously valued distribution specific to each analyte type. Notethat in this last step, the parameters of the imaging intensitydistributions are not part of the training process.

One key feature of this model is that it accounts for the spatialrelationship of analyte blobs within the vessel and also to each other,recognizing that point-wise image features (whether from histologyand/or radiology) is not the only source of information for experts todetermine plaque composition. While the model allows for the ability totrain without explicit histology-to-radiology registration, it couldalso be applied in situations where that registration is known. It isbelieved that statistically modeling the spatial layout ofatherosclerotic plaque components for classifying unseen plaques is anovel concept.

Example techniques for estimating vessel wall composition from CT or MRimages are further elaborated on in the following section. Inparticular, the methods may employ a multi-scale Bayesian analyticmodel. The basic Bayesian formulation is as follows:

${P\left( {A❘I} \right)} = {{\frac{{P\left( {A❘I} \right)} \cdot {P(A)}}{P(I)}{posterior}} = \frac{{likelihood} \cdot {prior}}{evidence}}$

In the context of the present disclosure, the hypothesis may be based ona multi-scale vessel wall analyte map, A, with observation combing fromCT or MR image intensity information I.

As depicted in FIG. 4 , the multi-scale vessel wall analyte map mayadvantageously include wall-level segmentation 410 (e.g., across-sectional slice of the vessel), blob-level segmentation 420 andpixel-level segmentation 430 (e.g., based on individual image pixels.E.g., A=(B,C) may be defined as a map of vessel wall class labels(similar to a graph with vertices B and edges C), wherein B is a set ofblobs (cross-sectionally contiguous regions of non-background wallsharing a label) and C is a set of blob couples or pairs. B_(b) may bedefined as a generic single blob where b∈[1 . . . n_(B)] is an index ofall blobs in A. B_(b) ^(a) is a blob with label a. For statisticalpurposes, the individual blob descriptor operator D_(B){ } is in anlow-dimensional space. C_(c) may be defined as a blob pair where c∈[1 .. . n_(B)(n_(B)−1)/2] is an index of all blob pairs in A. C_(c) ^(f,g)is a blob pair with labels f and g. For statistical purposes, the blobpair descriptor operator D_(C){ } is in a low-dimensional space. A(x)=amay be defined as the class label of pixel x where a∈{‘CALC’, ‘LRNC’,‘FIBR’, ‘IPH’, ‘background’ } (compositional characteristics). Inexemplary embodiments, I(x) is the continuous valued pixel intensity atpixel x. Within each blob, I(x) are modeled as independent. Note thatbecause the model is used to classify wall composition in 3Dradiological images, the word “pixel” is used to generically denote both2D pixels and 3D voxels

Characteristics of blob regions of like composition/structure mayadvantageously provide insights regarding the disease process. Eachslice (e.g., cross-sectional slice) of a vessel may advantageouslyinclude a plurality of blobs. Relationships between blobs may beevaluated in a pairwise manner. The number of blobs within across-section is modeled as a discrete random variable and may also beof quantifiable significance. At the slice-level of segmentation,relevant characteristics (e.g., biological properties/analytes) mayinclude a quantification of a total number of blobs and/or a number ofblobs of a particular structure/composition classification;relationships between the blobs, e.g., spatial relationships such asbeing closer to the interior. At the blob level of segmentation,characteristics of each blob, such as structural characteristics, e.g.,size and shape, as well as compositional characteristics, etc., may beevaluated serving as a biological properties/analytes. Finally, at apixel-level of segmentation, individual pixel level analysis may beperformed, e.g., based image intensity distribution.

Probability mapping of characteristics may be applied with respect tothe multi-scale vessel wall analyte map depicted in FIG. 4 . Theprobability mapping may advantageously establish a vector ofprobabilities for every pixel with components of the vector for theprobability of each class of analyte and one component for theprobability of background tissue. In example embodiments, sets ofprobability vectors may represent mutually exclusive characteristics.Thus, each set of probability vectors representing mutually exclusivecharacteristics will sum to 1. For example, in some embodiments, it maybe known that a pixel should fall into one and only one compositionalcategory (e.g., a single coordinate of a vessel cannot be both fibrousand calcified). Of particular note, the probability mapping does notassume independence of analyte class between pixels. This, is becauseneighboring pixels or pixels within a same blob may typically have sameor similar characteristics. Thus, the probability mapping accounts, asdescribed in greater detail herein, advantageously accounts fordependence between pixels.

f(A=α) may be defined as the probability density of map α. f(A) is theprobability distribution function over all vessel walls.f(D_(B){B^(a)}=β) is the probability density of descriptor vector β withlabel a. f(D_(B){B^(a)}) is the probability density function (pdf) ofblob descriptors with label a. There is a probability distributionfunction for each value of a. f(B)=Π f(D_(B){B^(a)}) f(D_(C){C^(f,g)}=γ)is the probability density of pairwise descriptor vector γ with labels fand g. f(D_(C){C^(f,g)}) is the probability density function (pdf) ofpairwise blob descriptors. There is a probability distribution functionfor each ordered pair f,g. Thus:f(C)Πf(D _(c) {C ^(a)})f(A)=f(B)f(C)=Πf(D _(b) {B ^(a)})Πf(D _(c) {C ^(a)})

P(A(x)=a) is the probability of pixel x having label a. P(A(x)) is theprobability mass function (pmf) of analytes (prevalence). It can beconsidered a vector of probabilities at a specific pixel x or as aprobability map for a specific class label value.Note that: f(A)=P(N)·f(C)·f(B)=P(N)·Πf(C _(c))·Πf(B _(b))

f(C_(c)=γ) is the probability density of pairwise descriptor vector γ.f(C_(c)) is the probability density function (pdf) of pairwise blobdescriptors. f(B_(b)=β) is the probability density of descriptor vectorβ. f(B_(b)) is the probability density function (pdf) of blobdescriptors. P(A(x)=a) is the probability of pixel x having label a.P(A(x)) is the probability mass function (pmf) of analytes (prevalencein a given map). It can be considered a vector of probabilities at aspecific pixel x or as a spatial probability map for a specific analytetype. P(A(x)=alI(x)=i) is the probability of analyte given the imageintensity that is our main goal to compute. P(I(x)=ilA(x)=a) is thedistribution of image intensities for a given analyte.

FIG. 5 depicts an exemplary pixel-level probability mass function as aset of analyte probability vectors. As noted above, the followingassumptions may inform the probability mass function: Completeness: inexample embodiments one may assume a sufficiently small pixel must fallinto at least one of the analyte classes (including a catch-all‘background’ category) and thus the sum of probabilities sums to 1.Mutual exclusivity: a sufficiently small pixel may be assumed to belongto only one class of analyte; if there are combinations (i.e.,spiculated calcium on LRNC background), then a new combination class canbe created in order to retain mutual exclusivity. Non-independence: eachpixel may be assumed to be highly dependent on its neighbors and theoverall structure of A.

An alternative view of the analyte map is as a spatial map ofprobability for a given analyte. At any given point during inference,analyte blobs can be defined using the full width half max rule. Usingthis rule, for each local maxima of probability for that analyte aregion is grown outward to a lower threshold of half the local maximavalue. Note that this 50% value is a tunable parameter. Spatialregularization of blobs can be done here by performing some curvatureevolution on probability maps in order to keep boundaries more realistic(smooth with few topological holes). Note that different possibleputative blobs of different analyte classes may in general have spatialoverlap because until one collapses the probabilities these representalternative hypotheses for the same pixel and hence the modifier‘putative’.

When iterative inference is terminated, there are several options forpresentation of the results. First, the continuously valued probabilitymaps can be presented directly to the user in one of several formsincluding but not limited to surface plots, iso-contour plots, or usingimage fusion similar to visualizing PET values as variation in hue andsaturation on top of CT. A second alternative is to collapse theprobability map at each pixel by choosing a single analyte label foreach pixel. This can be done most straightforwardly by choosing themaximum a posteriori value at each pixel independently, thus creating acategorical map which could be visualized by assigning a distinct colorto each analyte label and assigning either full or partial opacity ontop of the radiological image. Under this second alternative, the labelvalues might be assigned non-independently by resolving overlappingputative blobs based on a priority the probability of each blob. Hence,at a given pixel a lower priority analyte probability might be used forthe label if it belongs to a higher probability blob.

FIG. 6 illustrates a technique for computing putative analyte blobs. Inexample embodiments putative blobs may have overlapping regions. Thus,it may be advantageous to apply analytical techniques to segmentingpixels by putative blobs. For a probability of a given analyte the localmaxima in probability is determined. The full width half max rule maythen be applied to determine discrete blobs. At any given iteration ofinference, analyte blobs can be defined using the full width half maxrule. Find local maxima, then region grow with a lower threshold of0.5*max. (The 50% value could be a tunable parameter.) In someembodiments, spatial regularization of blobs may also be applied, e.g.,by performing some curvature evolution on probability maps in order tokeep boundaries smooth and avoid holes. Note that at this stagedifferent possible putative blobs of different analyte classes may, ingeneral, have spatial overlap because until probabilities are collapsedthese represent alternative hypotheses. Thus, an image-level analyte mapbe computed, e.g., based on a collapse of the probability map function.Notably, this collapse can be determined based on either the pixel-levelanalyte probability map, the putative blobs or a combination of both.With respect to the pixel-level analyte probability map, collapse can bedetermined by for each pixel, by choosing the label with maximumprobability A(x):=arg max a P(A(x)=a). This is similar to implementationViterbi algorithm. Basically, the highest probability for each set ofmutually exclusive probabilities vectors is locked in (e.g. with analytepriorities breaking possible ties). All other probabilities in the setmay then be set to zero. In some embodiments, probabilities forneighboring pixels/regions may be taken into account when collapsingdata on a pixel level. With respect to putative blob level collapse,overlapping putative blobs may be resolved. In some embodiments,prioritization can be based on blob probability density f(D₁{A_(a)^(b)}=d₁). Since higher probability blobs may change shape of overlappedlower probability blob this may impact analysis of blob levelcharacteristics. In example embodiments, the full range of probabilitiesmay be maintained rather than collapsing the data.

In order to model the relative spatial positioning of blobs within thevessel wall, an appropriate coordinate system can be chosen in order toprovide rotational-, translational-, and scale-invariance betweendifferent images. These invariances are important to the model becausethey allow the ability to train on one type of vessel (e.g., carotidswhere endarterectomy specimens are easily available) and apply the modelto other vessel beds (e.g., coronary where plaque specimens aregenerally not available) under the assumption that the atheroscleroticprocess is similar across different vessel beds. For tubular objects, anatural coordinate system follows from the vessel centerline wheredistance along the centerline provides a longitudinal coordinate andeach plane perpendicular to the centerline has polar coordinates ofradial distance and angle. However, due to the variability of vesselwall geometry, especially in the diseased patients, which one may aim toanalyze, an improved coordinate system may be utilized. The longitudinaldistance is computed in a way so that each 3D radiological image pixelis given a value, not just along the centerline or along interpolatedperpendicular planes. For a given plaque, the proximal and distal planesperpendicular to the centerline are each used to create an unsigneddistance map on the original image grid, denoted P(x) and D(x),respectively where x represents the 3D coordinates. The distance mapl(x)=P(x)/(P(x)+D(x)) represents the relative distance along the plaquewith a value of 0 at the proximal plane and 1 at the distal plane. Thedirection of the 1-axis is determined by ∇l(x).

Because the geometry of the wall may be significantly non-circular, theradial distance may be defined based on the shortest distance to theinner luminal surface and the shortest distance to the outer adventitialsurface. The expert-annotation of the histology images includes regionsthat define the lumen and the vessel (defined as the union of the lumenand vessel wall). A signed distance function can be created for each ofthese, L(x) and V(x), respectively. The convention is that the interiorof these regions is negative so that in the wall L is positive and V isnegative. The relative radial distance is computed asr(x)=L(x)/(L(x)−V(x)). It has a value of 0 at the luminal surface and 1at the adventitial surface. The direction of the r-axis is determined by∇r(x).

Because of the non-circular wall geometry, the normalized tangentialdistance may be defined as lying along iso-contours of r (and of l ifprocessing in 3D). The direction of the t-axis is determined by ∇r×∇l.The convention is that histology slices are assumed to be viewed lookingfrom the proximal to the distal direction so that positive l points intothe image. Note that unlike the others, t does not have a natural originsince it wraps onto itself around the vessel. Thus, one can define theorigin of this coordinate differently for each blob relative to thecentroid of the blob.

Another wall coordinate that is used is normalized wall thickness. Insome sense, this is a proxy for disease progression. Thicker wall isassumed to be due to more advanced disease. Assumption that statisticalrelationship of analytes changes with more advanced disease. Theabsolute wall thickness is easily calculated as w_(abs)(x)=L(x)−V(x). Inorder to normalize it to the range of [0-1], one may determine thatmaximum possible wall thickness when the lumen approaches zero size andis completely eccentric and near the outer surface. In this case themaximum diameter is the maximum Feret diameter of the vessel, D_(max).Thus, the relative wall thickness is computed asw(x)=w_(abs)(x)/D_(max).

The degree to which the aforementioned coordinates may or may not beused in the model is in part dependent on the amount of training dataavailable. When training data is limited, several options are available.The relative longitudinal distance may be ignored treating differentsections through each plaque as though they come from the samestatistical distribution. It has been observed that plaque compositionchanges along the longitudinal axis with more severe plaque appearancein the middle. However, instead of parameterizing the distributions byl(x), this dimension can be collapsed. Similarly, the relative wallthickness may also be collapsed. Observations have been made thatcertain analytes occur in “shoulder” regions of plaques where w(x) wouldhave a middle value. However, this dimension can also be collapsed untilenough training data is available.

As noted above, a vessel wall composition model may be utilized as theinitial hypothesis (e.g., at the prior P(A)). FIG. 7 depicts normalizedvessel wall coordinates for an exemplary vessel wall composition model.In the depicted model, 1 is relative longitudinal distance along vesseltarget from proximal to distal, which may be calculated, e.g., on anormalized the interval [0,1]. The longitudinal distance may be computedwith 2 fast marching propagations starting from proximal and from distalplanes to compute unsigned distance fields P and D wherein l=P/(P+D).l-axis direction is ∇l. As depicted, r is normalized radial distancewhich may also be calculated on a normalized interval [0,1] from luminalto adventitial surface. Thus, r=L/(L+(−V)) where L is lumen signeddistance field (SDF) and V is vessel SDF. r-axis direction is ∇r.Finally, t is normalized tangential distance which may be computed,e.g., on a normalized interval [−0.5,0.5]. Notably, in exampleembodiments there is may be no meaningful origin for the entire wall,only for individual analyte blobs (thus, t origin may be at blobcentroid). The tangential distance is computed along iso-contour curvesof l and of r. t-axis direction is ∇r×∇l.

FIG. 9 illustrates some complex vessel topologies which can be accountedfor using the techniques described herein. In particular, whenprocessing CT or MR in 3D, different branches may be advantageouslyanalyzed separately so that the relationship between analyte blobs inseparate branches are properly ignored. Thus, if a segmented view(cross-sectional slice) If includes more than one lumen, one can accountfor this by performing a watershed transform on r in order to split upwall into domains belonging to each lumen after which each domain may beseparately considered/analyzed.

As noted above, many of the coordinates and probability measurementsdescribed herein may be represented utilizing normalized scales therebypreserving scale invariance, e.g., between different sized vessels.Thus, the proposed model may advantageously be independent of absolutevessel size, under the assumption that a disease process is similar andproportional for different caliber vessels.

In some embodiments, the model may be configured to characterizeconcentric vs. eccentric plaque. Notably, a normalized all thicknessclose to 1 may indicate highly eccentric place. In further embodiments,inward vs. outward plaque characterization may be implemented. Notably,histological information on this characteristic is hindered bydeformation. Thus, in some embodiments, CT and training data may beutilized to establish an algorithm for determining inward vs. outwardplaque characterization.

As noted above, in example embodiments, non-imaging data, such ashistology data, may be utilized as a training set for establishingalgorithms linking image features to biological properties/analytes.There are however, some differences between the data types that need tobe addressed in ensuring a proper correlation. For example, thefollowing differences between histology and imaging may impact propercorrelation: Carotid endarterectomy (CEA) leaves adventitia and somemedia behind in patient CT or MR image analysis presumed to find outeradventitial surface. (See e.g., FIG. 8 depicting the margin between theplaque removed for the histology specimen relative to the outer vesselwall). Notably, scientific literature shows uncertainty of whethercalcification can occur in adventitia. The following techniques may beemployed to account for this difference. Histology can be dilatedoutward, e.g., based on an assumption that little to no analyte in thewall is left behind. Alternatively, Image segmentation can be erodedinward, e.g., based on knowledge of typical or particular margins left.For example, an average margin may be utilized. In some embodiment anaverage margin may be normalized a percentage of the overall diameter ofthe vessel. In further embodiments, histology may be used to mask theimaging (e.g., overlay, based on alignment criteria). In suchembodiments it may be necessary to apply one or more transformations tothe histology data to match proper alignment. Finally, in someembodiments, the difference may be ignored (which is equivalent touniform scaling of removed plaque to entire wall). While this may inducesome small error, presumably the wall left behind may be thin comparedto plaque in CEA patients.

Longitudinal differences may also exist between histological data (e.g.,a training set) and the imaging data as represented by the vessel wallcomposition model. In example embodiments, longitudinal distance may bemodeled/correlated explicitly. Thus, e.g., histology slice numbering(A-G for example) can be used to roughly determine position withinexcised portion of plaque. This approach, however, limits analysis withrespect to other slices without corresponding histology data. Thus,alternatively, in some embodiments, all histology slices may be treatedas arising from the same distribution. In example embodiments, somelimited regularization may still be employed along the longitudinaldirection.

As noted above, normalized wall thickness, in some sense is an imperfectproxy for disease progression. In particular, a thicker wall is assumedto be due to more advanced disease, e.g. based on an assumption thatstatistical relationship of analytes changes with more advanced disease.Normalized wall thickness may be calculated as follows: An absolute wallthickness T_(a) may be determined (in mm), e.g., computed asT_(a)=L+(−V) where L is lumen SDF, V is vessel SDF and D_(max) ismaximum Feret diameter of vessel (in mm). A relative wall thickness Tmay then be computed based on T=T_(a)/D_(max), e.g., on an interval[0,1], where 1 indicates thickest part of small lumen indicative ofcompletely eccentric plaque. In example embodiments, probabilities maybe conditioned based on wall thickness, e.g., so that the distributionof analyte blobs would depend on wall thickness. This advantageously maymodel differences in analyte composition over the course of diseaseprogression.

FIG. 10 depicts representing an exemplary analyte blob with adistribution of normalized vessel wall coordinates. In particular, theorigin oft is placed at blob centroid. (r,t) coordinates are a randomvector where the location/shape is fully represented by the jointdistribution of points within. This can be simplified by considering themarginal distributions (since radial and tangential shapecharacteristics seem relatively independent). Marginal distributions maybe calculated as projections along r and t (note that l and Tcoordinates can also be considered). Notably, the marginal distributionin the radial direction may advantageously represent/characterize theplaque growth in concentric layers (e.g., medial layer, adventitiallayer and intima layer.) Similarly, the marginal distribution in thetangential direction may advantageously represent a growth factor whichmay be indicative of the staging of the disease. In example embodiments,analyte blob descriptors can be computed based on the marginaldistributions. For example, on can take low order statistics on themarginal distributions (or use histograms or fit parametric probabilitydistribution functions).

In example embodiments, the following analyte blob descriptors may beused, e.g., to capture location, shape or other structuralcharacteristics of individual blobs:

-   -   Location in normalized vessel coordinates        -   Mostly with respect to r            -   e.g., in order to distinguish between shallow/deep                calcification        -   t-direction ignored; [optionally model l-direction]    -   Extent in normalized vessel coordinates        -   Intentionally avoiding the word ‘size’ which implies an            absolute measurement, whereas extent is a normalized value    -   Lopsidedness to represent degree of asymmetry in distribution        -   Clinical significance is unclear but it may help to            regularize shapes against implausible lopsided shapes    -   Alignment to represent confinement to parallel tissue layers        -   Analyte blobs seem to stay within radial layers            (iso-contours of r) quite well so this will help select            image processed shapes that are similar    -   Wall thickness where the blob is located        -   Thick (i.e., advanced) plaques assumed to have different            statistics than thin plaques

In some embodiments, pair-wise blob descriptors may also be utilized.For example:

-   -   Relative location        -   e.g., if fibrosis is on the lumen side of LRNC    -   Relative extent        -   e.g., how thick/wide is fibrosis relative to LRNC    -   Surroundedness        -   How much one marginal projection falls close to the middle            of the other        -   e.g., napkin ring sign or fibrosis around LRNC    -   Relative wall thickness        -   To represent degree of ‘shoulderness’ (shoulder would be            relatively less thick than central plaque body)

It is noted that higher order interactions (e.g., between three blobs orbetween two blobs and another feature), may also be implemented.However, consideration may be given to diminishing returns and traininglimitations.

The following are example quantifications of blob descriptors:

Individual Blob Descriptors

Location α_(r) = E[r] Extent β_(r) = Var[r] β_(t) = Var[t] Lopsidednessγ_(r) = | Skewness[r] | γ_(t) = | Skewness[t] | Alignment δ_(r) =Kurtosis[r] δ_(t) = Kurtosis[t] Thickness τ_(T) = E[T]

Pairwise Blob Descriptors

Relative location α_(rr) = E[r₂]-E[r₁] α_(tt) = E[t₂]-E[t₁] Relativeextent β_(rr) = Var[r₂]/Var[_(r1)] β_(tt) = Var[t₂]/Var[t₁]Surroundedness ε_(rr) = |α_(rr)| β_(rr) ε_(tt) = |α_(tt)| β_(tt)Relative thickness τ_(TT) = E[T₂]/E[T₁]

Notably, the set of descriptors (e.g., 8-12 descriptors) form a finiteshape space that a blob lives in. One can then look at the distributionof a population of blobs as a distribution in this finite space. FIG. 11depicts an exemplary distribution of blob descriptors. In exampleembodiments the distribution of blob descriptors may be computed overthe whole training set. In some embodiments, lower order statistics maybe utilized on individual blob descriptors (assuming independence),e.g., Location: E[α_(r)], Var[α_(r)]. In other embodiments, amulti-dimensional Gaussian (mean vector+covariance matrix) analysis maybe used to model the descriptors (e.g., wherein independence is notassumed). In further embodiments, if the distribution is non-normal itmay be modeled with density estimation techniques.

As noted above, one can also model a number of blobs per cross section(or the number of each class), e.g., η without regard to analyte classand η_(i) counting number in each analyte class. FIG. 14 depictsfrequency distribution of the total number of blobs for each histologyslide. A poison regression is applied as an overly. Note that theanalytic chart of FIG. 14 depicts the number of blobs per cross sectionN without regard to analyte class (number of blobs of each analyte typeis represented by B).

Summarizing the forgoing sections, in example embodiments, the overallvessel wall composition model may include the following:

-   -   Per-pixel analyte prior pmf        -   P(A(x)=a_(i))=ρ_(i)    -   Individual blob descriptors        -   B₁=(α_(r), β_(r), β_(t), γ_(r), γ_(t), δ_(r), δ_(t), τ_(T))        -   B₁˜N(μ₁,Σ₁)    -   Pairwise blob descriptors        -   C₂=(α_(rr), α_(tt), β_(rr), β_(tt), ε_(rr), ε_(tt), τ_(TT))        -   C₂˜N(μ₂,Σ₂)    -   Number of blobs        -   η˜Poisson(λ_(η))

wherein:

${{P\left( {{A(x)} = a_{i}} \right)} = \rho_{i}}{{f\left( A^{b} \right)} = {f\left( B_{1}^{b} \right)}}{{f(A)} = {{{P(\eta)} \cdot \left( {\underset{b \neq c}{\Pi}{f\left( C_{2}^{bc} \right)}} \right) \cdot \underset{b}{\Pi}}{f\left( A^{b} \right)}}}$

As noted above, an imaging model may serve as the likelihood (e.g.,P(I\A)) for the Bayesian analytic model. A maximum likelihood estimatemay then be determined. In example embodiments, this may be doneconsidering each pixel in isolation (e.g., without regard to the priorprobability of the structure in the model). Estimated analyte maps aretypically smooth only because images are smooth (which is why no priorsmoothing is typically performed). Independent pixel-by-pixel analysiscan be done, e.g., at least up to the point of accounting for scannerPSF. The imaging model is utilized to account for imperfect imagingdata. For example, imaging small components of plaque adds independentnoise on top of pixel values. Moreover, the partial volume effect andscanner PSF are well known as applying to small objects. Thus, given amodel (e.g., level set representation of analyte regions), simulating CTby Gaussian blurring with PSF is easy and fast. The imaging modeldescribed herein may also be applied to determine (or estimate) thedistribution of true (not blurred) densities of different analytes.Notably this cannot come from typical imaging studies since these willhave blurred image intensities. In some embodiments, wide variancescould be used to represent the uncertainty. Alternatively, distributionparameters could be optimized from training set but the objectivefunction would have to be based on downstream readings (of analyteareas), e.g., unless aligned histology data is available. FIG. 12depicts the exemplary model for imaging data (e.g., correlating betweena hidden (categorical) state (A(x)) and an observed (continuous) state(I(x)) whereby random (e.g., analyte density distribution (H(A(x))) anddeterministic (e.g., scanner blur *G(x)) noise factors are accountedfor. θ are the parameters of H (proportion & HU mean/variance of eachanalyte). θ=(τ₁,μ₁,σ₁, . . . ,τ_(N),μ_(N),σ_(N)) for N different analyteclasses assuming normal distributions. Note that 0 are patient specificand will be estimated in an expectation maximization (EM) fashion, e.g.,wherein analyte labels are the latent variables and the image isobserved data.

-   -   E-step: determine membership probabilities given current        parameters    -   M-step: maximize likelihood of parameters given membership        probabilities

FIG. 13 depicts a diagram of an example Markov model/Viterbi algorithmfor relating an observed state to a hidden state in an image model. Inparticular, the diagram depicts an observed state (gray) (observed imageintensity, I(x)) and a hidden state (white) (pure analyte intensity,H(A(x))) which can be modeled either with empirical histogram or withGaussian or boxcar probability distribution function. PSF of imagingsystem is modeled as Gaussian, G(x). Thus,I(x)=G(x)*H(A(x))

It is noted that a Viterbi-like algorithm could apply here butconvolution would replace emission probabilities H could be modeled asGaussian or uniform.

As noted above, one portion of the inference procedure is based uponexpectation maximization (EM). In a typical application of EM, datapoints are modeled as belonging to one of several classes, which isunknown. Each data point has a feature vector and for each class, thisfeature vector may be modeled with a parametric distribution such as amultidimensional Gaussian, represented by a mean vector and a covariancematrix. In the context of the model presented herein, a straightforwardEM implementation would work as follows:

${{L\left( {\theta;I} \right)} = {\prod\limits_{x = 1}^{N_{pixels}}{\sum\limits_{a = 1}^{N_{analytes}}{\tau_{a}{G\left( {{{I(x)};\mu_{a}},\sigma_{a}} \right)}{where}G{is}{Gaussian}{function}}}}}{\begin{matrix}{{L\left( {{\theta;A},I} \right)} =} & {{p\left( {I,{A❘\theta}} \right)} = {\prod\limits_{x = 1}^{N_{pixels}}{\sum\limits_{a = 1}^{N_{analytes}}{\delta_{a,{A(x)}}\tau_{a}{G\left( {{{I(x)};\mu_{a}},\sigma_{a}} \right)}}}}} \\\text{} & {{where}\delta{is}{Kronecker}{delta}} \\ = & {\exp\left\{ {\sum\limits_{x = 1}^{N_{pixels}}{\sum\limits_{a = 1}^{N_{analytes}}{\delta_{a,{A(x)}}\left\lbrack {{\ln\tau_{a}} - \frac{\ln\left( {2{\pi\sigma}_{a}^{2}} \right)}{2} -} \right.}}} \right.} \\\text{} & \left. \left. {}\frac{\left( {{I(x)} - \mu_{a}} \right)^{2}}{2\sigma_{a}^{2}} \right\rbrack \right\}\end{matrix}}{{T_{j,x}^{(t)}:={P\left( {{{A(x)} = {{j❘I} = {I(x)}}};\theta^{(t)}} \right)}} = {\frac{\tau_{j}^{(t)}{G\left( {{{I(x)};\mu_{j}^{(t)}},\sigma_{j}^{(t)}} \right)}}{\sum\limits_{a = 1}^{N_{analytes}}{\tau_{a}^{(t)}{G\left( {{{I(x)};\mu_{a}^{(t)}},\sigma_{a}^{(t)}} \right)}}}\left( {{membership}{probabilities}} \right)}}{\begin{matrix}{{Q\left( {\theta ❘\theta^{(t)}} \right)} = {E\left\lbrack {\ln{L\left( {{\theta;A},I} \right)}} \right\rbrack}} \\{= {E\left\lbrack {\ln{\prod\limits_{x = 1}^{N_{pixels}}{L\left( {{\theta;{I(x)}},{A(x)}} \right)}}} \right\rbrack}} \\{= {\sum\limits_{x = 1}^{N_{pixels}}{E\left\lbrack {\ln{L\left( {{\theta;{I(x)}},{A(x)}} \right)}} \right\rbrack}}} \\{= {\sum\limits_{a = 1}^{N_{analytes}}{\sum\limits_{x = 1}^{N_{pixels}}{T_{a,x}^{(t)}\left\lbrack {{\ln\tau_{a}} - {1\frac{\ln\left( {2{\pi\sigma}_{a}^{2}} \right)}{2}} - \frac{\left( {{I(x)} - \mu_{a}} \right)^{2}}{2\sigma_{a}^{2}}} \right\rbrack}}}}\end{matrix}}{\tau^{({t + 1})} = {\underset{\tau}{argmax}\left\{ {\sum\limits_{a = 1}^{N_{analytes}}\left( {\left\lbrack {\sum\limits_{x = 1}^{N_{pixels}}T_{a,x}^{(t)}} \right\rbrack\ln\tau_{a}} \right)} \right\}}}{\tau_{j}^{({t + 1})} = {\frac{1}{N_{pixels}}{\sum\limits_{x = 1}^{N_{pixels}}T_{j,x}^{(t)}}}}{\left( {\mu_{j}^{({t + 1})},\sigma_{j}^{({t + 1})}} \right) = {\underset{\mu,\sigma}{argmax}\left\{ {\sum\limits_{a = 1}^{N_{analytes}}\left( {\left\lbrack {\sum\limits_{x = 1}^{N_{pixels}}T_{a,x}^{(t)}} \right\rbrack\left\lbrack {{{- 1}\frac{\ln\left( {2{\pi\sigma}_{a}^{2}} \right)}{2}} - \frac{\left( {{I(x)} - \mu_{a}} \right)^{2}}{2\sigma_{a}^{2}}} \right\rbrack} \right)} \right\}}}{\mu_{j}^{({t + 1})} = {{\frac{\sum\limits_{x = 1}^{N_{pixels}}{T_{j,{I(x)}}^{(t)}{I(x)}}}{\sum\limits_{x = 1}^{N_{pixels}}T_{j,{I(x)}}^{(t)}}\sigma_{j}^{({t + 1})}} = \frac{\sum\limits_{x = 1}^{N_{pixels}}{T_{j,{I(x)}}^{(t)}\left( {{I(x)} - \mu_{j}^{({t + 1})}} \right)}^{2}}{\sum\limits_{x = 1}^{N_{pixels}}T_{j,{I(x)}}^{(t)}}}}$

The main problem with this simple model is that it doesn't code anyhigher order structure to the pixels. There is no prior probabilityassociated with more realistic arrangements of pixels. Only taudetermines the proportion of analyte classes. Thus, once can use the tauvariable to insert in the blob prior probability model, in particular atthe step of updating membership probabilities.

Thus, a modified Bayesian inference procedure may be applied with a muchmore sophisticated Bayesian prior. In the basic EM implementation, thereis no real prior distribution. The variable tau represents the a priorirelative proportion of each class but even this variable is unspecifiedand estimated during the inference procedure. Thus, there is no a prioribelief about the distribution of classes in the basic EM model. In ourmodel, the model prior is represented by the multi-scale analyte model.Tau becomes a function of position (and other variables), not just aglobal proportion.

$\begin{matrix}{{L\left( {{\theta;A},I} \right)} =} & {{f\left( {I,{A❘\theta}} \right)} = {{f\left( {{I❘A},\theta} \right)} =}} \\\text{} & {{f(A)}{\prod\limits_{x = 1}^{N_{pixels}}{G\left( {{{I(x)};\mu_{A(x)}},\sigma_{A(x)}} \right)}}} \\ = & {{f(A)}\exp\left\{ {{\sum\limits_{x = 1}^{N_{pixels}}{- \frac{\ln\left( {2{\pi\sigma}_{a}^{2}} \right)}{2}}} - \frac{\left( {{I(x)} - \mu_{a}} \right)^{2}}{2\sigma_{a}^{2}}} \right\}}\end{matrix}\begin{matrix}{{Q\left( {\theta ❘\theta^{(t)}} \right)} =} & {E\left\lbrack {\ln{L\left( {{\theta;A},I} \right)}} \right\rbrack} \\ = & {E\left\lbrack {{\ln{f(A)}} + {\sum\limits_{x = 1}^{N_{pixels}}{- \frac{\ln\left( {2{\pi\sigma}_{a}^{2}} \right)}{2}}} - \frac{\left( {{I(x)} - \mu_{a}} \right)^{2}}{2\mu_{a}^{2}}} \right\rbrack} \\ = & {{E\left\lbrack {\ln{f(A)}} \right\rbrack} +} \\\text{} & {\sum\limits_{a = 1}^{N_{analytes}}{\sum\limits_{x = 1}^{N_{pixels}}{T_{a,x}^{(t)}\left\lbrack {{{- 1}\frac{\ln\left( {2{\pi\sigma}_{a}^{2}} \right)}{2}} - \frac{\left( {{I(x)} - \mu_{a}} \right)^{2}}{2\sigma_{a}^{2}}} \right\rbrack}}}\end{matrix}{\left( {\mu_{j}^{({t + 1})},\sigma_{j}^{({t + 1})}} \right) = {\underset{\mu,\sigma}{argmax}\left\{ {\sum\limits_{a = 1}^{N_{analytes}}\left( {\left\lbrack {\sum\limits_{x = 1}^{N_{pixels}}T_{a,x}^{(t)}} \right\rbrack\left\lbrack {{{- 1}\frac{\ln\left( {2{\pi\sigma}_{a}^{2}} \right)}{2}} - \frac{\left( {{I(x)} - \mu_{a}} \right)^{2}}{2\sigma_{a}^{2}}} \right\rbrack} \right)} \right\}}}{\mu_{j}^{({t + 1})} = {{\frac{\sum\limits_{x = 1}^{N_{pixels}}{T_{j,{I(x)}}^{(t)}{I(x)}}}{\sum\limits_{x = 1}^{N_{pixels}}T_{j,{I(x)}}^{(t)}}\sigma_{j}^{({t + 1})}} = \frac{\sum\limits_{x = 1}^{N_{pixels}}{T_{j,{I(x)}}^{(t)}\left( {{I(x)} - \mu_{j}^{({t + 1})}} \right)}^{2}}{\sum\limits_{x = 1}^{N_{pixels}}T_{j,{I(x)}}^{(t)}}}}$

The membership probability function is defined as follows:

${{f\left( {I,{A❘\theta}} \right)} = {{{f(A)}{f\left( {{I❘A},\theta} \right)}} = {{f(A)}{\prod\limits_{x = 1}^{N_{pixels}}{G\left( {{{I(x)};\mu_{A(x)}},\sigma_{A(x)}} \right)}}}}}{{f\left( {{A❘I},\theta} \right)} = {\frac{1}{Z}{f(A)}{f\left( {{I❘A},\theta} \right)}}}{{P\left( {{{A(x)} = {{j❘{I(x)}} = i}},\theta} \right)} = {\frac{1}{Z}{P\left( {{A(x)} = j} \right)}{f\left( {{{I(x)} = {{i❘{A(x)}} = j}},\theta} \right)}}}{T_{j,x}^{(t)}:={P\left( {{{A(x)}^{(t)} = {{j❘{I(x)}} = i}},\theta} \right)}}{T_{j,x}^{(t)}:=\frac{{P\left( {{A(x)}^{(t)} = j} \right)}{G\left( {{{I(x)};\mu_{j}^{(t)}},\sigma_{j}^{(t)}} \right)}}{\sum\limits_{a = 1}^{N_{analytes}}{{P\left( {{A(x)}^{(t)} = a} \right)}{G\left( {{{I(x)};\mu_{a}^{(t)}},\sigma_{a}^{(t)}} \right)}}}\left( {{membership}{probabilities}} \right)}{T_{j,x}^{(t)}:=\frac{1}{Z}{\sum\limits_{models}\left\lbrack {{P\left( {{A(x)} = j} \right)}{P\left( {{{{{I(x)} - i}❘{A(x)}} = j},\theta} \right)}} \right\rbrack}}{:=\frac{1}{Z}{\sum\limits_{\alpha \in {models}}{{P\left( {A = \alpha} \right)}{P\left( {{A(x)} = j} \right)}{P\left( {{{I(x)} = {{i❘{A(x)}} = j}},\theta} \right)}}}}{:=\frac{1}{Z}{\sum\limits_{\alpha \in {models}}{{P(N)}\Pi{f\left( C_{c} \right)}\Pi{f\left( B_{b} \right)}{P\left( {{A(x)} = j} \right)}{P\left( {{{I(x)} = {{i❘{A(x)}} = j}},\theta} \right)}}}}$

The inference algorithm is as follows. At each step of iteration, themembership probability map is initialized to zero so that all classeshave zero probability. Then for all possible model configurations, themembership probability map may be added to as follows:T _(j,x) ^((t)) +=P(N ^((t)))Πf(C _(c) ^((t)))Πf(B _(b)^((t)))P(A(x)^((t)) =j)P(I(x)=i|A(x)^((t)) =j,θ)

Finally, the probability vector may be normalized at each pixel in themembership probability map to restore the completeness assumption.Advantageously one can iterate over all model configurations. This isdone by sequentially considering values for N from 0 to a relatively lowvalue, for instance 9, at which point extremely few sections have everbeen observed to have as many blobs. For each value of N, one canexamine different putative blob configurations. The putative blobs maybe thresholded to a small number (N) based on their individual blobprobabilities. Then, all of the permutations of N blobs are considered.Thus, one can simultaneously considering all of the most likely blobconfigurations and weighting each model by its prior probability. Thisprocedure is obviously an approximate inference scheme since the fullspace of multi-scale model configurations may not be considered. One canassume, however, that by considering the most likely (in terms of both Nand blobs), a good approximation is achieved. This procedure alsoassumes that the weighted average of the most likely configurationsprovides a good estimate at each individual pixel. Another alternativeis to perform a constrained search of model configurations and selectthe highest likelihood model as the MAP (maximum a posteriori) estimate.

Further exemplary statistical models (e.g., the posterior P(A\I)) arealso described herein. In a CT angiography the following information maybe available:

-   -   Intensity        -   CT Hounsfield units or MR intensities        -   Possibly other imaging features    -   Position relative to anatomy        -   Where in the plaque a pixel is    -   Neighboring pixels        -   E.g., for smoothing contours through level sets

Posterior probability may be computed as:P(A|I)∝(P(I|A)·P(A)

Thus, the following image information may influence analyte probability,Ai(x)

-   -   I(x) is observed image intensity (possibly a vector)    -   T(x) is observed relative wall thickness from image segmentation    -   F(x) are CT image features    -   S(x) are features of vessel wall shape (e.g., luminal bulge)

In some embodiments a Metropolis-Hastings like approach may be utilized.In other embodiments a maximum a posteriori approach may be applied.

The following are example algorithmic possibility for a statisticalanalysis model. In some embodiments, the model may utilize Beliefpropagation (AKA max sum, max product, sum product messaging). Thus, forexample a Viterbi (HMM) type approach may be utilized, e.g., wherein,hidden states are the analyte assignments, A, observed states are theimage intensities, I. This approach may advantageously find a MAPestimate may be argmax P(AII). In some embodiments a soft output Viterbialgorithm (SOVA) may be utilized. Note that reliability of each decisionmay be indicated by difference between chosen (survivor) path anddiscarded path. Thus, this could indicate reliability of each pixelanalyte classification. In further example embodiments aforward/backward Baum-Welch (HMM) approach may be utilized. For example,one can compute most likely state at any point in time but not the mostlikely sequence (see Viterbi).

Another possible technique is the Metropolis-Hastings (MCMC) approach,e.g., wherein one repeatedly samples A and weights by likelihood andprior. In some embodiments, a simple MRF version for sampling may beutilized. Note that it may be particularly advantageous to sample theposterior directly. In example embodiments, one can build up per-pixelhistograms of analyte class.

Other algorithm possibilities include applying a Gibbs Sampler,Variational Bayes (similar to EM), Mean field approximation, a Kalmanfilter, or other techniques.

As noted above, in some embodiments an Expectation Maximization (EM)posterior approach may be utilized. Under this approach, observed data Xis the imaging values, unknown parameters θ are due to the analyte map(but not including analyte probabilities) and latent variable Z is theanalyte probability vector. One key feature of this approach is that itenables iterating between estimating class membership (Z) and modelparameters (θ) since they each depend on each other. However, since theanalyte map separates out analyte probabilities, the approach may bemodified such that the current class membership doesn't have toinfluence the model parameters (since these are learned this during atraining step). Thus, EM basically learning the model parameters as ititerates through the current data. Advantageously, exemplaryimplementation of the EM approach iteratively compute maximum likelihoodbut assumes a flat prior.

Techniques are also provided herein for representing longitudinalcovariance. Due to wide spacing of histology slices (e.g., 4 mm),sampling may not faithfully capture the longitudinal variation inanalytes. However, 3D image analysis is typically performed andpresumably there is some true longitudinal covariance. The problem isthat histological information typically isn't provided for longitudinalcovariance. Nonetheless the exemplary statistical models disclosedherein may reflect a slow variation in longitudinal direction.

In some embodiments, a Markov model/chain may be applied. FIG. 15depicts exemplary implantation of a 1D Markov chain for Text/DNA.Conventionally, when applied to images in MRF Markov chains are typicalas low order as possible. A higher order chain may be advantageous,however, due to conditional independence (Markov property). Otherwisethe data may be too scrambled to be of value. This is demonstrated bythe 1D sampling of an exemplary Markov chain as applied to text:

-   -   Uniform probability sampling output:        -   earryjnv anr            jakroyvnbqkrxtgashqtzifzstqaqwgktlfgidmxxaxmmhzmgbya            mjgxnlyattvc rwpsszwfhimovkvgknlgddou nmytnxpvdescbg k            syfdhwqdrj jmcovoyodzkcofmlycehpcqpuflje            xkcykcwbdaifculiluyqerxfwlmpvtlyqkv    -   0-order Markov chain output:        -   ooyusdii eltgotoroo tih ohnnattti gyagditghreay nm            roefnnasos r naa euuecocrrfca ayas el s yba anoropnn laeo            piileo hssiod idlif beeghec ebnnioouhuehinely neiis            cnitcwasohs ooglpyocp h trog l    -   1^(st) order Markov chain output:        -   icke inginatenc blof ade and jalorghe y at helmin by hem            owery fa st sin r d n cke s t w anks hinioro e orin en s ar            whes ore jot j whede chrve blan ted sesourethegebe inaberens            s ichath fle watt o    -   2^(nd) order Markov chain output:        -   he ton th a s my caroodif flows an the er ity thayertione            wil ha m othenre re creara quichow mushing whe so mosing            bloack abeenem used she sighembs inglis day p wer wharon the            graiddid wor thad k    -   3^(rd) order Markov chain output:        -   es in angull o shoppinjust stees ther a kercourats allech is            hote ternal liked be weavy because in coy mrs hand room him            rolio and ceran in that he mound a dishine when what to            bitcho way forgot p

FIG. 16 depicts an example first order Markov chain for a textprobability table. Note that such tables are exponentially sized interms of order:

D=order of Markov chain

N=number of letters

Size=N^(D)

Thus, higher order leads to problems with dimensionality. Advantageouslyhistology samples have a very high resolution. However, since histologysamples are not statistically independent, this may lead to overfittingas later described in greater detail. In general, the more conditionaldependence that is modeled, the more predictive the model can be.

In example embodiments, a 2D Markov random field (MRF) may be used forpixel values instead of a 1D sequence such as for letters. FIG. 17depicts conditional dependence of a first pixel (black) based on itsneighboring pixels (gray). In example embodiments cliques may make usesymmetry to reduce the number of dependencies in half. In someembodiments, the values of pixels could be simple image intensities orcould be probability values for classification problems. Problems existwith typical MRF use. Conventional MRF almost always is limited to thenearest neighbor pixels providing conditional dependence which greatlyreduces the specificity of the probability space represented; usuallyjust black/white blobs for very general purpose segmentation/filtering;extremely short range dependencies. However, whereas pixels are highlydiscretized a blob just missing one pixel and falling in the next maycompletely change the probability distribution. Thus, a real imagestructure is much more continuous than is typically accounted for usingMRF.

For this reasons the systems and methods of the present disclosure mayadvantageously utilize an inference procedure, e.g., a Bayes type ruleof Posterior a Likelihood×Prior (P(A/I) α P(I/A)×P(A)). Using acrossword type analogy, the inference procedure implemented by thesystems and methods of the subject application is a bit like trying toOCR a crossword puzzle from a noisy scan. Knowledge (even imperfectknowledge of several squares may help inform an unknown square in thecrossword puzzle. Efficiently is improved even more by considering bothvertical and horizontal direction simultaneously. In exampleembodiments, the inference procedure may be heuristic. For example, onecan initialize with uninformed prior, then, solve the easier ones first,which gives you clues about the harder ones which are solved later.Thus, relatively easy to detect biological properties such as densecalcium may inform the existence of other harder to detect analytes suchas lipids. Each step of the inference procedure may narrow theprobability distributions for unsolved pixels.

As noted above a high order Markov chain is preferable to obtain usabledata. The disadvantage of utilizing a higher order Markov approach isthat there may not be enough data to inform the inference process. Inexample embodiments, this issue may be addressed by utilizing densityestimation methods such as Parzen windowing or utilizing krigingtechniques.

To form an inference procedure, one may initialize with unconditionalprior probabilities of analytes and then use a highest level of evidenceto start narrowing down probabilities. For example, in some embodiments,an uncertain width may be associate with each analyte probabilityestimate. In other embodiments, closeness to 1/N may represent suchuncertainty.

Notably, the term “Markov” is used loosely herein since the proposedMarkov implementations are not memoryless but rather are explicitlytrying to model long range (spatial) dependencies.

Because the CT resolution is low compared to histology and plaqueanatomy, in some embodiments it may be preferable to utilize acontinuous space (time) Markov model rather than discrete space (time).This may work well with the level set representation of probability mapssince they naturally work well with sub-pixel interpolation. Discreteanalyte states makes the model a discrete space model. However, if onerepresents continuous probabilities rather than analytepresence/absence, then it becomes a continuous space model.

Turning to lung based applications, table 4 below depicts exemplarybiological properties/analytes which may utilized with respect to ahierarchical analytics framework for such applications.

TABLE 5 Biologically-objective measurands Supported by lung basedapplications Category Description Readings Units/Categories Size Thesize of the Volume (lesion, mm{circumflex over ( )}3 lesion solidportion, ground-glass portion) Longest mm diameter and perpendicular(lesion, solid portion, ground-glass portion) Shape/ Overall shape Shapesphericity (unitless: Margin of the lesion round = 1, oval ~0.5, andline = 0)/lobulated— descriptions of irregular/cavitary, its borderspeculation, notch/cut which may indicate certain Margin Tumor marginscale cancers or (HU) Tumor margin diseases window (HU/mm) (possiblyTopology Euler Number including fibrotic scarring) Solidity Mean Volume% solid % development of Lesion (C/T of cell types or ratio) lackthereof Volume % % that make up ground-glass of the lesion Lesion(differentiation, Solid density g/ml organization) Ground glass g/mldensity Mass of solid g Mass of ground g glass Heterogeneity Covarianceand SD (variation of g/ml development of solid density) cell types or SD(variation of g/ml lack thereof ground glass) that make up PatternNonsolid or ground- the lesion glass opacity (pure (differentiation,GGN)/perifissural/ organization) part-solid (mixed GGN)/solid Solidportion Radial intensity pattern distribution 1^(st) and 2^(nd) orderstatistics (Central/central with ring/diffuse/peripheral) Spatial NSM(non-spatial coherence methods); SGLM (texture, (spatial gray-level“clumpiness”, methods) e.g., localized Haralick; fractal heterogeneity)analysis (FA): Lacunarity, average local variance, variance of localvariance, average of local average; filters & transforms (F&T) e.g.,Gabor Invasive Measure of Pleural contact mm Potential Lesion's length(AKA invasive extent arch distance) or potential Pleural contactunitless extent length-to- maximum lesion diameter Pleural Displacementfrom Involvement expected location Lobe Location Upper/middle/lowerlobe//right/left Lobe centrality unitless (1 = lobe center, 0 = lobe )boundary Airway category Involvement/air bronchogram VascularDilated/rigid/ changes convergent/tortuous Calcification Response toVolume mm{circumflex over ( )}3 injurious agent Volume % of %(dystrophic) or Lesion caused by Distribution Central/peripheral/deranged diffuse metabolism Pattern amorphous/punctuate/ (metastatic)reticular/popcorn/ laminated Cell Measures of Uptake SUV (unitless), % IMetabolism cell metabolism D/g Glycolytc volume <each non- ChangePairwise In units of measurand categorical assessed arithmetic measurandbetween as few difference above> as 2 but Pairwise ratio unitlessarbitrarily Pairwise days/weeks/months many doubling time timepointsPolynomial fit coefficients Non-arithmetic change assessment withregistration, e.g., vascular changes <each non- Assessed over TotalTumor mm{circumflex over ( )}3 categorical multiple targets Burdenmeasurand according to Tumor Number unitless above> response MultilobarTrue/false criteria, e.g., Lymph Node category RECIST, status WHO, etc.Metastasis category Response category

In particular, systems may be configured to detect lung lesions. Thus,an exemplary system may be configured for whole lung segmentation. Insome embodiments, this may involve use of minimum curvature evolution tosolve juxtapleural lesion problems. In some embodiments, the system mayimplement lung component analysis (vessel, fissure, bronchi, lesionetc.). Advantageously a Hessian filter may be utilized to facilitatelung component analysis. In some embodiments lung component analysis mayfurther include pleural involvement, e.g., as a function of fissuregeometry. In further embodiments, attachment to anatomic structures mayalso be considered. In addition to lung component analysis, separateanalysis of ground glass vs. solid stated may also be applied. This mayinclude determination of geometric features, such as volume, diameter,sphericity, image features, such as density and mass, and fractalanalysis.

Fractal analysis may be used to infer lepidic growth patterns. In orderto perform fractal analysis on very small regions of interest, ourmethod adaptively modifies the support for convolution kernels to limitthem to the region of interest (i.e., lung nodule). Intersectingvessels/bronchi as well as non-lesion feature may be masked out for thepurposes of fractal analysis. This is done by applying IIR Gaussianfilters over masked local neighborhoods and normalizing with IIR blurredbinary masking. In some embodiments, fractal analysis may furtherinclude determining lacunarity (based on variance of the local mean).This may be applied with respect to lung lesions, subparts of lesions.In example embodiments, IIR Gaussian filters or circular neighborhoodsmay be applied. In some embodiments IIR may be utilized to computevariance. Average of local variance (AVL) may also be computed, e.g., asapplied to lung lesions. Likewise, a variance of local variance may becalculated.

In example embodiments, both lesion structure and composition may becalculated. Advantageously calculating lesion structure may utilize fullvolumetry of thin sections thereby improving on calculating sizemeasurement change. Measurements such as sub-solid and ground glassopacity (GGO) volume may also be determined as part of assessing lesionstructure. Turning to lesion composition, tissue characteristics such asconsolidation, invasion, proximity and perfusion may be calculated e.g.,thereby reducing false positive rate relative to conventional analytics.

With reference now to FIG. 18 , a further exemplary hierarchicalanalytics framework 1800 for the systems of the present disclosure isdepicted. FIG. 18 may be understood as an elaboration of FIG. 1elucidating greater detail with respect to exemplary intermediateprocessing layers of the hierarchical inference system. Advantageouslythe hierarchical inferences still flow from imaging data 1810 tounderlying biological information 1820 to clinical disease 1830.Notably, however, the framework 1800 includes multiple levels of datapoints for processing imaging data in order to determine biologicalproperties/analytes. At a pre-processing level 1812, physicalparameters, registrations transformations and region segmentations maybe determined. This preprocessed imaging information may then beutilized to extract imaging features at the next level of data points1814 such as intensity features, shape, texture, temporalcharacteristics, and the like. Extracted image features may nextutilized at level 1816 to fit one or more biological models to theimaged anatomy. Example models may include a Bayes/Markov net lesionsubstructure, a fractal growth model, or other models such as describedherein. The biological model may advantageously act as a bridge forcorrelating imaging features to underlying biologicalproperties/analytes at level 1822. Example biologicalproperties/analytes include anatomic structure, tissue composition,biological function, gene expression correlates, and the like. Finally,at level 1824 the biological properties/analytes may be utilized todetermine clinical findings related to the pathology including, e.g.,related to disease subtype, prognosis, decision support and the like.

FIG. 19 is an example application of phenotyping purpose in directingvascular therapy, using the Stary plaque typing system adopted by theAHA as an underlay with in vivo determined types shown in coloroverlays. The left panel shows an example that labels according to thelikely dynamic behavior of the plaque lesion based on its physicalcharacteristics, and the right panel illustrates an example which usesthe classification result for directing patient treatment. An examplemapping is [‘I’,‘II’,‘III’,‘IV’,‘V’,‘VI’,‘VII’,‘VIII’] yieldingclass_map=[Subclinical, Subclinical, Subclinical, Subclinical, Unstable,Unstable, Stable, Stable]. This method is not tied to Stary, e.g., theVirmani system [‘Calcified nodule’, ‘CTO’, ‘FA’, ‘FCP’, ‘Healed PlaqueRupture’, ‘PIT’, ‘IPH’, ‘Rupture’, ‘TCFA’, ‘ULC’] has been used withclass_map=[Stable, Stable, Stable, Stable, Stable, Stable, Unstable,Unstable, Unstable, Unstable], and other typing systems may yieldsimilarly high performance. In example embodiments, the systems andmethods of the present disclosure may merge disparate typing systems,the class map may be changed, or other variations. For FFR phenotypes,values such as normal or abnormal may be used, and/or numbers may beused, to facilitate comparison with physical FFR for example.

FIG. 20 is example for a different disease, e.g., lung cancer. In thisexample, the subtypes of masses are determined so as to direct the mostlikely beneficial treatment for the patient based on the manifestphenotype.

CNNs are expected to perform better than readings-vector classificationbecause CNNs contain filters which extract spatial context which isn'tincluded in (only) analyte area measurements. It may be practical to usea CNN despite the reduced training set because

-   -   1) there are relatively few classes corresponding to        significantly different treatment alternatives (rather than        being fully granular as might be done in research assays        necessitating ex vivo tissue), e.g. three phenotypes for the        classification problem, three risk levels for the outcome        prediction/risk stratification problem, so the problem is        generally easier.    -   2) the processing of analyte regions into false color regions,        e.g., by level sets or other algorithm classes, performs a        substantial portion of the image interpretation by generating        the segmentations and presenting the classifier with a        simplified, but considerably enriched data set. Measurable        pipeline stages reduce the dimensionality of the data (reducing        the complexity of the problem that the CNN must solve) while        also providing verifiable intermediate values which can increase        confidence in the overall pipeline.    -   3) re-formatting the data using a normalized coordinate system        removes noise variation due to variables that do not have a        substantial impact on the classification, e.g., vessel size in        the plaque phenotyping example.

To test this idea a pipeline was built consisting of three stages:

-   -   1) semantic segmentation to identify which regions of the        biomass fall into certain classes    -   2) spatial unwrapping to convert the vein/artery cross section        into a rectangle, and    -   3) a trained CNN to read the annotated rectangles and identify        which class (stable or unstable) it pertains to.

Without loss of generality, example systems and methods described hereinmay apply spatial unwrapping (for example, training and testing CNNswith (unwrapped dataset) and without (donut dataset) spatialunwrapping). Unwrapping was observed to improve the validation accuracy

Semantic Segmentation and Spatial Unwrapping:

First, the image volume is preprocessed. This may include targetinitialization, normalization, and other pre-processing such asdeblurring or restoring to form a region of interest containing aphysiological target that is to be phenotyped. Said region is a volumecomposed of cross sections through that volume. Body site is eitherautomatically determined or is provided explicitly by user. Targets forbody sites that are tubular in nature are accompanied with a centerline.Centerlines, when present, can branch. Branches can be labelled eitherautomatically or by user. Generalizations on centerline concept may berepresented for anatomy that is not tubular but which benefit by somestructural directionality, e.g., regions of a tumor. In any case, acentroid is determined for each cross section in the volume. For tubularstructures this will be the center of the channel, e.g., the lumen of avessel. For lesions this will be the center of mass of the tumor.

FIG. 21 is representative of an exemplary image pre-processing step, inthis case deblurring or restoring using a patient-specific point spreaddetermination algorithm to mitigate artifacts or image limitations thatresult from the image formation process that may decrease the ability todetermine characteristics predictive of the phenotype. The figuredemonstrates a portion of the radiology analysis application analysis ofa plaque from CT. Shown here are the deblurred or restored images thatare a result of iteratively fitting a physical model of the scannerpoint spread function with regularizing assumptions about the truelatent density of different regions of the image. This figure isincluded so as to illustrate that a variety of image processingoperations may be performed so as to aid in the ability to performquantitative steps, and in no way to indicate that this method is neededfor the specific invention in this disclosure but rather being exemplaryof steps which may be taken to improve overall performance.

The (optionally deblurred or restored) image is represented in aCartesian data set where x is used to represent how far from centroid, yrepresents a rotational theta, and z represents the cross section. Onesuch Cartesian set will be formed per branch or region. When multiplesets are used, a “null” value will be used for overlapping regions, thatis, each physical voxel will be represented only once across the sets,in such a way as to geometrically fit together. Each data set will bepaired with an additional data set with sub-regions labelled byobjectively verifiable tissue composition (see, e.g., FIG. 36 ). Examplelabels for vascular tissue can be lumen, calcification, LRNC, etc.Example labels for lesions could be necrotic, neovascularized, etc.These labels can be validated objectively, e.g. by histology (see, e.g.,FIG. 37 ). Paired data sets will used as input to a training step tobuild a convolutional neural network. Two levels of analysis aresupported, one at an individual cross-section level, optionally wherethe output varies continuously across adjacent cross-sections, and asecond at the volume level (where individual cross-sections may bethought of as still frames, and the vessel tree traversal could beconsidered as analogous to movies).

Exemplary CNN Design:

AlexNet is a CNN, which competed in the ImageNet Large Scale VisualRecognition Challenge in 2012. The network achieved a top-5 error of15.3%. AlexNet was designed by the SuperVision group, consisting of AlexKrizhevsky, Geoffrey Hinton, and Ilya Sutskever at U Toronto at thetime. AlexNet was trained from scratch to classify an independent set ofimages (not used in training and validation steps during the networktraining). For the unwrapped data an AlexNet style network with 400×200pixel input was used, and the donut network is AlexNet style with280×280 pixel input (roughly the same resolution but different aspectratio). All of the convolutional filter values were initialized withweights taken from AlexNet trained on the ImageNet dataset. While theImageNet dataset is a natural image dataset, this simply serves as aneffective method of weight initialization. Once training begins, allweights are adjusted to better fit the new task. Most of the trainingschedule was taken directly from the open source AlexNet implementation,but some adjustment was needed. Specifically, the base learning rate wasreduced to 0.001 (solver.prototxt) and the batch size was reduced to 32(train_val.prototxt) for both the AlexNet-donut and AlexNet-unwrappednetworks. All models were trained to 10,000 iterations and were comparedto snapshots when trained till just 2,000 iterations. While a more indepth study on overfitting could be done, it was generally found thatboth training and validation error decreased between 2 k and 10 kiterations.

Alternative featurizers (prefixes) could include:

-   -   ResNet—https://arxiv.org/abs/1512.03385    -   GoogLeNet—https://www.cs.unc.edu/˜wliu/papers/GoogLeNet.pdf    -   ResNext—https://arxiv.org/abs/1611.05431    -   ShuffleNet V2—https://arxiv.org/abs/1807.11164    -   MobileNet V2—https://arxiv.org/abs/1801.04381

Run-time optimizations such as frame-to-frame redundancy betweencross-sections (sometimes referred to as “temporal” redundancy, but inour case, being a form of inter-cross-section redundancy) could beleveraged to save on computation (e.g.,http://arxiv.org/abs/1803.06312). Many optimizations for training orinference may be implemented.

In example test implementations, AlexNet was trained to classify anindependent set of images between two categories of clinicalsignificance, e.g., ‘unstable’ plaques and ‘stable’ plaques, based onhistology ground truth plaque types of V and VI, while the latterincludes plaque types VII and VIII following the industry de-factostandard plaque classification nomenclature accepted by the AmericanHeart Association (AHA), and on a related but distinct typing system byVirmani

Without loss of generality, in illustrated examples, both total accuracyand a confusion matrix were be utilized to assess performance. Thisformalism was based on the notion of computing four possibilities in abinary classification system: true positives, true negatives, falsepositives and false negatives. In example embodiments, other outcomevariables can be used, however, for example, one can utilize sensitivityand specificity as outcome variables, or the F1 score (the harmonic meanof precision and sensitivity). Alternatively, an AUC characteristic canbe computed for a binary classifier. Furthermore, classifiers need notbe binary based. For example, in some embodiments, classifiers may sortbased on more than two possible states.

Dataset Augmentation:

Physician annotated data is expensive, so it is desirable toartificially increase medical datasets (e.g., for use in training and/orvalidation). Two different augmentation techniques were used in exampleembodiments described herein. Donuts were horizontally flipped randomly,as well as rotated to a random angle from 0 to 360. The resultingrotated donut was then cropped to the range in which the donut waspresent, and then padded with black pixels to fill the image to have asquare aspect ratio. The result was then scaled to the 280×280 size andsaved to a PNG.

The unwrapped dataset was augmented by randomly horizontally flipping,and then “scrolled” by a random number of pixels in the range from 0 tothe width of the image. The result was then scaled to the 400×200 sizeand saved to a PNG.

Both datasets were increased by a factor of 15, meaning that the totalnumber of images after augmentation is 15 times the original number.Class normalization was implemented, meaning that the final dataset hasroughly the same number of images pertaining to each class. This isimportant as the original number of images for each class might bedifferent, thus biasing the classifier to the class with the largernumber of images in the training set.

Without loss off generality, each radiologist who performed theannotations can use an arbitrary number of tissue types.

FIG. 22 illustrates an exemplary application that used to demonstrateaspects of the present invention, in this case being for classificationof atherosclerotic plaque phenotype classification (using specificsubject data by way of example). Different colors represent differenttissue analyte types with dark gray showing the otherwise normal wall.The figure illustrates the result of ground truth annotation of tissuecharacteristics that are indicative of plaque phenotype as well as thespatial context of how they present in a cross section taken orthogonalto the axis of the vessel. It also illustrates a coordinate system thathas been developed in order to provide a common basis for analysis of alarge number of histological cross sections. Grid lines added todemonstrate coordinate system (tangential vs. radial distance) andoverlaid on top of color-coded pathologist annotations. An importantaspect of this is that data sets of this kind may be used efficiently indeep learning approaches because they simplify the information using arelatively simpler false color image in place of a higher-resolutionfull image but without losing spatial context, e.g., to have a formalrepresentation for such presentations as “napkin ring sign”,juxtaluminal calcium, thin (or thick) caps (spacing between LRNC andlumen), etc.

FIG. 23 illustrates tangential and radial direction variable internallyrepresented using unit phasors and here, phasor angle shown coded ingray scale, which exemplifies the use of normalized axes for tubularstructures relevant to the vascular and other pathophysiology associatedwith such structures (e.g., the gastro-intestinal tract). Note that thevertical bar from black to white is a purely arbitrary boundary due tothe gray scale encoding, and the normalized radial distance has a valueof 0 at the luminal boundary and value of 1 at the outer boundary.

FIG. 24 illustrates an exemplary overlay of radiology analysisapplication generated annotations from CTA (unfilled color contours) ontop of pathologist generated annotations from histology (solid colorregions). An example aspect of the systems and methods presented hereinis that the contours form in vivo non-invasive imaging can be used withthe classification scheme so as to determine phenotype non-invasively,where the classifier is trained on known ground truth. Specifically,filling in the contours which are shown unfilled in this figure (so asto not obscure the relationship with the ex vivo annotation for thisspecific section which is provided to show the correspondence) createsinput data for the classifier.

FIG. 25 demonstrates a further step of data enrichment, specifically,utilizing the normalized coordinate system to avoid non-relevantvariation associated with the wall thickness and radial presentation.Specifically, the “donut” is “unwrapped” while retaining the pathologistannotations. The left panel illustrates pathologist region annotationsof a histological slice of a plaque after morphing to convert the cutopen “C” shape back into the in vivo “O” shape of the intact vesselwall. Horizontal axis is the tangential direction around the wall.Vertical axis is the normalized radial direction (bottom is luminalsurface, top is outer surface). Also note that the finer granularity ofthe pathologist annotations has been collapsed to match the granularityintended for extraction by the in vivo radiology analysis application(e.g., LRNC, CALC, IPH). The right panel illustrates the comparableunwrapped radiology analysis application annotations. Axes and colorsare the same as the pathologist annotations.

FIG. 26 represents the next refinement relevant to the plaquephenotyping example. Working from the unwrapped formalism, luminalirregularity (as results, for example from ulceration or thrombus) andlocal varying wall thickening are represented. The light grey at bottomrepresents the lumen (added in this step so as to represent that luminalsurface irregularity) and the black used in the prior step is nowreplaced with dark grey, to represent the varying wall thickening. Blacknow represents area outside of the wall entirely.

FIG. 27 represents an additional example, so as to include, for example,intra-plaque hemorrhage and/or other morphology aspects as needed (usingspecific subject data by way of example). The left panel shows the donutrepresentation and the right panel the unwrapped with the luminalsurface and localized wall thickening represented.

Example CNNs tested included CNNs based on AlexNet and Inceptionframeworks.

AlexNet Results:

In example embodiments tested, the convolutional filter values wereinitialized with weights taken from AlexNet trained on the ImageNetdataset. While the ImageNet dataset is a natural image dataset, thissimply serves as an effective method of weight initialization. Oncetraining begins, all weights are adjusted to better fit the new task.

Most of the training schedule was taken directly from the open sourceAlexNet implementation, but some adjustment was needed. Specifically,the base learning rate was reduced to 0.001 (solver.prototxt) and thebatch size was reduced to 32 (train_val.prototxt) for both thealexnet-donut and alexnet-unwrapped networks.

All models were trained to 10,000 iterations and were compared tosnapshots when trained till just 2,000 iterations. While a more in depthstudy on overfitting could be done, it was generally found that bothtraining and validation error decreased between 2 k and 10 k iterations.

A brand new AlexNet network model was trained from scratch for 4 (four)different combinations of ground-truth results of two leadingpathologists, two different ways of processing images (see above), aswell as unwrapped images and donut images. The results are listed inFIG. 28 . Each dataset variation had its training data augmented by 15×with class normalization enabled. A network was trained on thisaugmented data and then was tested on the corresponding un-augmentedvalidation data corresponding to that variation. For the unwrapped dataan AlexNet style network with 400×200 pixel input was used, and thedonut network is AlexNet style with 280×280 pixel input (roughly thesame resolution but different aspect ratio). Note that in testembodiments the dimensions of the conventional layers as well as thefully connected layers were changed. Thus, the network in the AlexNettest embodiments can be described as a five convolutional layer, threefully connected layer network. Without loss of generality, here are somehigh-level conclusions that are illustrated from these results:

-   -   1) With the exception of the WN_RV dataset, it does indeed seem        to be that the unwrapped data is easier for the data to analyze        as it receives higher validation accuracy across the board    -   2) The non-normalized data is demonstrated to be more        representative as anticipated.    -   3) In regards to the WN_RV dataset, the original idea was to        pool WN and RV truth data to see the compatibility of the typing        systems and the degree to which sets may be merged. In doing so,        significant differences were observed in the WN vs RV data. The        original intention of the WN_RV experiments was to pool training        data from multiple pathologists to see if the information        contributed to efficacy. Instead degradation rather than        improvement was observed. This was determined to be because of        variations in color scheme which impeded such pooling of data.        Thus, one can consider normalizing the color scheme to enable        pooling.

Exemplary Alternative Network: Inception:

Transfer-learning re-training of an Inception v3 CNN was started withthe Aug. 8, 2016 version of the network uploaded on the TensorFlow sitefor public use. The network was trained for 10,000 steps. Training andValidation sets were normalized in number of images via imageaugmentation so both sub-sets amounted to the same number of annotatedimages. All other network parameters were taken to be at their defaultvalues.

Pre-trained CNNs can be used to classify imaging features using theoutput from the last convolution layer, which is a numeric tensor withdimensionality of 2048×2 in the case of the Google Inception v3 CNN. Wethen train an SVM classifier to recognize the object. This process isnormally performed on the Inception model after a transfer-learning andFine-Tuning steps in which the model initially trained on the ImageNet2014 dataset has its last, softmax layer removed and re-trained torecognize the new categories of images.

Alternative Embodiments

FIG. 29 provides an alternative example, for phenotyping potentiallycancerous lung lesions. The left-most panel indicates the outlines of asegmented lesion, with pre-processing to separate out into solid vs.semi-solid (“ground glass) sub regions. The middle panels indicate itslocation in the lung, and the right-most panel shows it with false coloroverlay. In this case, the 3-dimensional nature of the lesion is likelyconsidered significant, so instead of processing 2-D cross-sectionsseparately, techniques such as video interpretation from computer visionmay be applied for the classifier input data set. In fact, processingmultiple cross-sections sequentially, as if in a “movie” sequence alonga centerline, can generalize these methods for tubular structures.

Another generalization is where the false colors are not selected from adiscrete palette but instead have continuous values at pixel or voxellocations. Using the lung example, FIG. 30 shows a set of features,sometimes described as so-called “radiomics” features that can becalculated for each voxel. Such a set of values may exist in arbitrarynumber of pre-processed overlays and be fed into the phenotypeclassifier.

Other alternative embodiments include using change data, for example ascollected from multiple timepoints, rather than (only) data from asingle timepoint. For example, if the amount or nature of a negativecell type increased, it may be said to be a “progressor” phenotype, vs.a “regressor” phenotype for decreases. The regressor might be, forexample, due to response to a drug. Alternatively, if the rate of changefor, say, LRNC is rapid, this may imply a different phenotype. Theextension of the example to use delta values or rates of change isobvious to one skilled in the art.

As an additional alternative embodiment, non-spatial information, suchas which are derived from other assays (e.g., lab results), ordemographics/risk factors, or other measurements taken from theradiological image, may be fed into the final layers of the CNN tocombine the spatial information with non-spatial information. Also,localized information such as use of a pressure wire with readings atone or more certain locations along a vessel from a reference such as abifurcation or ostia, by inference of full 3D coordinates at imaging maybe determined.

Whereas the focus of these examples has been on phenotypeclassification, similar approaches may be applied to the problem ofoutcome prediction, as a further embodiment of this invention.

Example Implementations

Systems and methods of the present disclosure may advantageouslycomprise a pipeline consisting of multiple stages. FIG. 34 provides afurther example implementation of a hierarchical analytics framework.Biological properties are identified and quantified by semanticsegmentation to identify biological properties singly or in combination,in the example application lipid-rich necrotic core, cap thickness,stenosis, dilation, remodeling ratio, tortuosity (e.g., entrance andexit angles), calcification, IPH, and/or ulceration, which isrepresented numerically as well as in enriched data sets with spatialunwrapping to convert the vein/artery cross-section into a rectangle,and then medical conditions e.g., ischemia-causing fractional flowreserve FFR, high-risk phenotype HRP, erosion and/or risk stratificationtime to event (TTE) using trained CNN(s) to read the enriched data setsto identify and characterize the condition. Images are collected of thepatient, the raw slice data is used in a set of algorithms to measurebiological properties that may be objectively validated, these are inturn formed as enriched data sets to feed one of more CNNs, in thisexample where results are forward and back-propagated using recurrentCNNs to implement constraints or creates continuous conditions (such asa monotonically decreasing fractional flow reserve from proximal todistal throughout the vessel tree, or constant HRP value in a focallesion, or other constraints). Ground truth data for HRP may exist asexpert pathologist determined plaque types at given cross-sections,having been determined ex vivo. Ground truth data for FFR may be fromphysical pressure wire, with one or more measured values, and networktraining for locations along the centerline proximal of a givenmeasurement being constrained to be greater than or equal to themeasured value, locations distal being constrained to be less than orequal, and when two measurements on the same centerline are known, thatthe values between the two measured values be constrained within theinterval.

These properties and/or conditions may be assessed at a given point intime and/or change across time (longitudinal). Without loss ofgenerality, other embodiments performing similar steps, either in plaquephenotyping or in other applications, would be embodiments of theinvention.

In example implementations, biological properties can include one ormore of the following:

-   -   Angiogenesis    -   Neovascularization    -   Inflammation    -   Calcification    -   Lipid-deposits    -   Necrosis    -   Hemorrhage    -   Ulceration    -   Rigidity    -   Density    -   Stenosis    -   Dilation    -   Remodeling Ratio    -   Tortuosity    -   Flow (e.g., of blood in channel)    -   Pressure (e.g., of blood in channel or one tissue pressing        against another)    -   Cell types (e.g., macrophages)    -   Cell alignment (e.g., of smooth muscle cells)    -   Shear stress (e.g., of blood in channel)

Analysis can include determining one or more of quantity, degree and/orcharacter for each of the aforementioned biological properties.

Conditions that can be determined based on the biological properties mayinclude one or more of:

-   -   Perfusion/ischemia (as limited) (e.g., of brain or heart tissue)    -   Perfusion/infarction (as cut off) (e.g., of brain or heart        tissue)    -   Oxygenation    -   Metabolism    -   Flow reserve (ability to perfuse) e.g., FFR(+) vs. (−) and/or        continuous number    -   Malignancy    -   Encroachment    -   High-risk plaque e.g., HRP(+) vs. (−) and/or labelled phenotype    -   Erosion e.g., Erosion(+) vs. (−) and/or labelled phenotype    -   Risk stratification (whether as probability of event, or time to        event) (e.g., MACCE, mentioned explicitly)

Validation in the form of truth bases can include the following:

-   -   Biopsy    -   Expert tissue annotations form excised tissue (e.g.,        endarterectomy or autopsy)    -   Expert phenotype annotations on excised tissue (e.g.,        endarterectomy or autopsy)    -   Physical pressure wire    -   Other imaging modalities    -   Physiological monitoring (e.g., ECG, SaO2, etc.)    -   Genomic and/or proteomic and/or metabolomics and/or        transcriptomic assay    -   Clinical outcomes

Analysis can be both at a given point in time as well as longitudinal(i.e., change across time).

Exemplary System Architecture:

FIG. 31 illustrates a high-level view of the users and other systemsthat interact with an analytics platform, as per the systems and methodsof the present disclosure. Stakeholders of this view include SystemAdministrators, Support Technicians, which have Interoperability,Security, Failover & Disaster Recovery, Regulatory concerns.

The platform can be deployed in two main configurations; on-premises, orremote server (FIG. 32 ). The platform deployment may be a stand-aloneconfiguration (Left Upper), on-premises server configuration (LeftLower), or remote server configuration (Right). The on-premisesdeployment configuration can have two sub-configurations; desktop onlyor rackmount. In the remote configuration, the platform may be deployedon a HIPAA compliant data center. Clients access that API server over asecure HTTP connection. Clients can be desktop or tablet browsers. Nohardware except for the computers running the web browsers is deployedon the customer site. The deployed server may be on a public cloud or anextension of the customer's private network using a VPN.

An exemplary embodiment is comprised of a client and a server. Forexample, FIG. 33 illustrates a client as a C++ application and theserver as a Python application. These components interact using HTML5.0, CSS 5.0 and JavaScript. Wherever possible open standards are usedfor interfaces including but not limited to; HTTP(S), REST, DICOM,SPARQL, and JSON. Third party libraries are also used as shown in thisview which shows the primary pieces of the technology stack. Manyvariations and different approaches may be understood by people skilledin the art.

Various embodiments of the above-described systems and methods may beimplemented in digital electronic circuitry, in computer hardware,firmware, and/or software. The implementation can be as a computerprogram product (i.e., a computer program tangibly embodied in aninformation carrier). The implementation can, for example, be in amachine-readable storage device and/or in a propagated signal, forexecution by, or to control the operation of, data processing apparatus.The implementation can, for example, be a programmable processor, acomputer, and/or multiple computers.

A computer program can be written in any form of programming language,including compiled and/or interpreted languages, and the computerprogram can be deployed in any form, including as a stand-alone programor as a subroutine, element, and/or other unit suitable for use in acomputing environment. A computer program can be deployed to be executedon one computer or on multiple computers at one site.

Method steps can be performed by one or more programmable processorsexecuting a computer program to perform functions of the invention byoperating on input data and generating output. Method steps can also beperformed by and an apparatus can be implemented as special purposelogic circuitry. The circuitry can, for example, be a FPGA (fieldprogrammable gate array) and/or an ASIC (application specific integratedcircuit). Modules, subroutines, and software agents can refer toportions of the computer program, the processor, the special circuitry,software, and/or hardware that implements that functionality.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor receives instructions and data from a read-only memory or arandom access memory or both. The essential elements of a computer are aprocessor for executing instructions and one or more memory devices forstoring instructions and data. Generally, a computer can include, can beoperatively coupled to receive data from and/or transfer data to one ormore mass storage devices for storing data (e.g., magnetic,magneto-optical disks, or optical disks).

Data transmission and instructions can also occur over a communicationsnetwork. Information carriers suitable for embodying computer programinstructions and data include all forms of non-volatile memory,including by way of example semiconductor memory devices. Theinformation carriers can, for example, be EPROM, EEPROM, flash memorydevices, magnetic disks, internal hard disks, removable disks,magneto-optical disks, CD-ROM, and/or DVD-ROM disks. The processor andthe memory can be supplemented by, and/or incorporated in specialpurpose logic circuitry.

To provide for interaction with a user, the above described techniquescan be implemented on a computer having a display device. The displaydevice can, for example, be a cathode ray tube (CRT) and/or a liquidcrystal display (LCD) monitor. The interaction with a user can, forexample, be a display of information to the user and a keyboard and apointing device (e.g., a mouse or a trackball) by which the user canprovide input to the computer (e.g., interact with a user interfaceelement). Other kinds of devices can be used to provide for interactionwith a user. Other devices can, for example, be feedback provided to theuser in any form of sensory feedback (e.g., visual feedback, auditoryfeedback, or tactile feedback). Input from the user can, for example, bereceived in any form, including acoustic, speech, and/or tactile input.

The above described techniques can be implemented in a distributedcomputing system that includes a back-end component. The back-endcomponent can, for example, be a data server, a middleware component,and/or an application server. The above described techniques can beimplemented in a distributing computing system that includes a front-endcomponent. The front-end component can, for example, be a clientcomputer having a graphical user interface, a Web browser through whicha user can interact with an example implementation, and/or othergraphical user interfaces for a transmitting device. The components ofthe system can be interconnected by any form or medium of digital datacommunication (e.g., a communication network). Examples of communicationnetworks include a local area network (LAN), a wide area network (WAN),the Internet, wired networks, and/or wireless networks.

The system can include clients and servers. A client and a server aregenerally remote from each other and typically interact through acommunication network. The relationship of client and server arises byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other.

Packet-based networks can include, for example, the Internet, a carrierinternet protocol (IP) network (e.g., local area network (LAN), widearea network (WAN), campus area network (CAN), metropolitan area network(MAN), home area network (HAN)), a private IP network, an IP privatebranch exchange (IPBX), a wireless network (e.g., radio access network(RAN), 802.11 network, 802.16 network, general packet radio service(GPRS) network, HiperLAN), and/or other packet-based networks.Circuit-based networks can include, for example, the public switchedtelephone network (PSTN), a private branch exchange (PBX), a wirelessnetwork (e.g., RAN, Bluetooth, code-division multiple access (CDMA)network, time division multiple access (TDMA) network, global system formobile communications (GSM) network), and/or other circuit-basednetworks.

The computing device can include, for example, a computer, a computerwith a browser device, a telephone, an IP phone, a mobile device (e.g.,cellular phone, personal digital assistant (PDA) device, laptopcomputer, electronic mail device), and/or other communication devices.The browser device includes, for example, a computer (e.g., desktopcomputer, laptop computer) with a World Wide Web browser (e.g.,Microsoft® Internet Explorer® available from Microsoft Corporation,Mozilla® Firefox available from Mozilla Corporation). The mobilecomputing device includes, for example, a Blackberry®, iPAD®, iPhone® orother smartphone device.

Whereas many alterations and modifications of the disclosure will nodoubt become apparent to a person of ordinary skill in the art afterhaving read the foregoing description, it is to be understood that theparticular embodiments shown and described by way of illustration are inno way intended to be considered limiting. Further, the subject matterhas been described with reference to particular embodiments, butvariations within the spirit and scope of the disclosure will occur tothose skilled in the art. It is noted that the foregoing examples havebeen provided merely for the purpose of explanation and are in no way tobe construed as limiting of the present disclosure.

Although the present disclosure has been described herein with referenceto particular embodiments, the present disclosure is not intended to belimited to the particulars disclosed herein; rather, the presentdisclosure extends to all variations and generalizations thereof thatwould be apparent to a person of ordinary skill in the art includingthose within the broadest scope of the appended claims.

The invention claimed is:
 1. A method for computer aided determinationof omics of a pathology using an enriched radiological dataset, themethod comprising: receiving a radiological dataset for a patient,wherein the radiological dataset is obtained non-invasively; enrichingthe dataset by performing analyte measurement and/or classification ofone or more of: (i) anatomic structure, (ii) shape or geometry or (iii)tissue characteristic, type, or character, with objective validation fora set of analytes relevant to a pathology, wherein the analytemeasurement and/or classification of anatomic structure, shape, orgeometry and/or tissue characteristic, type, or character includessemantic segmentation to identify and classify regions of interest inthe radiological dataset, wherein the regions of interest are identifiedwith respect to cross-sections of a pathology-specific structure in theradiological dataset; and using a machine learned classificationapproach based on known ground truths to process the enriched datasetand determine an omic profile for the pathology.
 2. The method of claim1, wherein enriching the dataset further includes spatialtransformations of the dataset to accentuate biologically-significantspatial context.
 3. The method of claim 1 wherein the omic is a genomic,proteomic, metabolomic, transcriptomic or any combination thereof. 4.The method of claim 1, wherein enriching the dataset includes both (i)semantic segmentation to identify and classify regions of interest incross-sections of a pathology-specific structure in the radiologicaldataset to produce an annotated dataset and (ii) spatially transformingthe annotated dataset with respect to the cross-sections of apathology-specific structure to produce a pathology-appropriatetransformed dataset.
 5. The method of claim 1, wherein an image volumein the radiological dataset is preprocessed to form a region of interestcontaining a physiological target, lesion, and/or set of lesions that isto be analyzed.
 6. The method of claim 5, wherein the region of interestand/or the physiological target, lesion, and/or set of lesions are atleast one of (i) automatically determined from the radiological datasetor (ii) identified by a user from the radiological dataset.
 7. Themethod of claim 5, wherein the region of interest includes one or morecross sections, each composed of projections through that volume.
 8. Themethod of claim 5, wherein the pre-processing the image volume includesdeblurring or restoring using a patient-specific point spreaddetermination algorithm to mitigate artifacts or image limitations thatresult from the image formation process.
 9. The method of claim 1,wherein the machine learned classification approach is use of a trainedconvolutional neural network (CNN), wherein the CNN is based on arefactoring of AlexNET, Inception, CaffeNet, or other open source orcommercially available framework.
 10. The method of claim 1, wherein thedataset is enriched by visually using different colors to representdifferent analyte sub-regions.
 11. The method of claim 1, whereindataset enrichment includes ground truth annotation of analytesub-regions as well as providing a spatial context of how such analytessub-regions present in cross-section.
 12. The method of claim 11,wherein the spatial context provides a common basis for analysis ofenhanced dataset relative to histological cross-sections.
 13. The methodof claim 12, wherein the spatial context includes providing a coordinatesystem based on polar coordinates relative to a centroid of eachcross-section.
 14. The method of claim 1, wherein dataset enrichmentincludes ground truth annotation of analyte sub-regions using ex vivoclassification independent of or in conjunction with image-basedclassification based on a common spatial context between theradiological dataset and ex vivo data.
 15. The method of claim 14,wherein enriched dataset is visualized using different colors torepresent different analyte sub-regions, wherein colors for visualizingthe enhanced dataset are selected to correspond to colors utilized inthe ex vivo data.
 16. The method of claim 15, wherein the enricheddataset includes a visual overlay of ex-vivo data over the radiologicaldata.
 17. The method of claim 1, wherein the radiological datasetincludes computed tomography (CT), dual energy computed tomography(DECT), spectral computed tomography (spectral CT), computed tomographyangiography (CTA), cardiac computed tomography angiography (CCTA),magnetic resonance imaging (MRI), multi-contrast magnetic resonanceimaging (multi-contrast MRI), ultrasound (US), positron emissiontomography (PET), intra-vascular ultrasound (IVUS), optical coherencetomography (OCT), near-infrared radiation spectroscopy (NIRS), and/orsingle-photon emission tomography (SPECT) diagnostic images.
 18. Themethod of claim 17, wherein enriching the dataset includes using imagedeblurring or restoring is used to identify lesions of interest andextract pathology composition quantitatively.
 19. The method of claim17, wherein enriching the dataset includes spatially transformingcross-sectional segmented images into an ‘pathology-appropriatetransformed’ reference frame.
 20. The method of claim 1, furthercomprising using a machine learned classification approach based onknown ground truths to process the enriched dataset and determine apredictive outcome related to the pathology.